Z-sight joins robot action generation with latent world modeling
Maven Robotics and USC researchers introduced Z-sight, a robot foundation model that couples action generation to future visual tokens predicted in its VLM's native feature space.
Researchers affiliated with Maven Robotics and the University of Southern California introduced Z-sight, a robot foundation model that predicts future visual tokens in its vision-language model’s native feature space as it generates actions.
In the version-one preprint, the authors report an 81.9% average success rate on the SIMPLE humanoid benchmark and 73.3% across six physical Unitree G1 tasks. The findings come from the authors’ own evaluations and have not been independently replicated or peer reviewed in the sources examined.
Z-sight adds explicit foresight without reconstructing future video pixels or using a separate frozen encoder as its prediction target. Its 3.18-billion-parameter mixture-of-transformers combines a 2.22-billion-parameter Qwen3.5-2B vision-language backbone with separate 480-million-parameter foresight and action experts. The foresight expert predicts the visual tokens that the model’s evolving vision encoder would produce for a future observation. The action expert can attend to that prediction as it forms.
At inference, the two experts start with Gaussian noise, then update the future tokens and a 30-step action chunk together over 10 shared Euler steps. The model does not decode the tokens back into pixels. For training, an exponential-moving-average copy of the vision encoder supplies stop-gradient future targets, while the future-prediction loss is blocked from updating the vision-language backbone. The researchers say this setup keeps perception and prediction aligned while limiting representation collapse and the loss of pretrained knowledge.
The authors describe a three-stage training recipe. Pre-training uses 1,977 hours of egocentric human video. Mid-training uses about 2,750 hours of cross-embodiment robot data with human-video replay, bringing the pre- and mid-training source pool to 4,731 hours. Post-training then uses demonstrations from the target robot.
On SIMPLE, the reported 81.9% result spans 12 teleoperated tasks at three difficulty levels, putting it 9.6 percentage points above GR00T N1.7’s 72.3%. The paper says each method was evaluated in three passes, with 10 episodes for every task and difficulty level in each pass.
In the physical G1 evaluation, Z-sight completed 44 of 60 trials, or 73.3%, compared with 53.3% for Fast-WAM. Each method received 10 trials on each of six tasks from matched starting configurations. The paper does not report confidence intervals for those real-world rates.
The researchers also measured a median 109.5 milliseconds to generate a 30-step action chunk on a single RTX 4090, excluding image capture, network transport and low-level control. GR00T N1.7 was faster in that test at 60.0 milliseconds; three other systems in the comparison were slower. The paper also identifies a planning limitation: Z-sight predicts the future from observations alone rather than conditioning foresight on candidate actions, so it cannot yet compare possible action outcomes at test time.
More news

MuJoCo 4.0.1 corrects solver bug from major release

RECLAIM uses robot attention feedback to counter adversarial vision patches

IFR ranks South Korea first in 2026 robot-density release
