MBZUAI uses generated wrist views to help robots see through occlusion

Researchers at Mohamed bin Zayed University of Artificial Intelligence developed a method that turns an external camera view into a virtual wrist-camera image. The generated image is fed to a robot controller when no physical wrist camera is available at deployment, preserving close-range visual cues in settings where a wrist-mounted camera could be impractical or occluded.

In a controlled benchtop test described in the research paper, a Unitree Z1 arm detached plastic strawberries in 3 of 10 trials with only an external view. It succeeded in 6 of 10 trials with a synthesized in-hand view and 8 of 10 with a real wrist-camera view. Each configuration covered just 10 target positions under fixed indoor lighting, so the result is preliminary rather than evidence of field-ready performance.

The system starts with an RGB image from a fixed external camera and the calibrated relative pose between that camera and the desired wrist viewpoint. A ZeroNVS diffusion model, adapted to robot-manipulation scenes with Low-Rank Adaptation, uses those inputs to synthesize the missing in-hand image.

The generated image then enters the same dual-view visuomotor policy used with a real wrist camera. The policy processes the external image and the in-hand image through separate encoders, combines their features with robot-state information, and predicts an action trajectory. The experiments primarily used a Flow Matching Policy.

That substitution applies only at test time. Training the view generator still requires paired external and real wrist-camera images with their relative camera poses, while training the controller also requires real images from both viewpoints. The method therefore removes the physical wrist camera from deployment, not from data collection and model training.

The researchers also evaluated six simulated manipulation tasks from RoboMimic and MimicGen. For the synthesized-view condition, they first ran an external-view policy, then reran its failed trials with the generated wrist view enabled using the same 50 seeds. The protocol measures rescued failures; it is not a fresh full-trial comparison and does not show whether synthesized images could introduce new failures.

The current generator is not real time. The paper reports about 16.5 seconds to synthesize one frame on an NVIDIA RTX 4060 Ti, preventing the demonstrated implementation from continuously updating the virtual wrist view in a closed-loop controller. The authors point to faster generation, including model distillation, as future work.