NVIDIA releases Cosmos3-H-Dreams for real-time surgical simulation research

NVIDIA researchers have released Cosmos3-H-Dreams, a generative simulator that produces future endoscopic video from an initial surgical image and supplied robot commands. The research system is specialized for cholecystectomy scenes recorded with CMR Surgical’s Versius robot, and NVIDIA says it is not validated for patient care.

The release includes four-step and two-step model checkpoints plus source code and examples. NVIDIA also links to guidance for adapting the model to synchronized video and compatible controls. That guide describes a reference recipe rather than an executable end-to-end distillation pipeline: it says causal teacher-forcing and Self-Forcing training are not yet supported in the referenced framework, and the current loader does not provide the model’s 28-dimensional CMR control adapter.

Cosmos3-H-Dreams does not decide how the robot should move. It accepts a 28-dimensional command stream covering both robot arms—including pose and gripper information, energy controls, thumbstick axes and thumbstick buttons—and generates what the endoscopic view might look like after those actions. NVIDIA trained the model on selected windows containing two visible, engaged instruments: a left Cadiere grasper and a right monopolar hook. Camera commands were limited during sample selection, narrowing the setting demonstrated by the release.

The technical step is putting a causal, few-step video generator into a live control loop. NVIDIA says it first adapted a bidirectional teacher over 24-frame and 72-frame horizons, then distilled it into a student that generates short blocks as actions arrive. Self-Forcing exposes that student to its own generated history during training, while FlashDreams supplies the accelerated serving layer and an adapted FadeMem module consolidates the growing key-value cache during longer rollouts.

The reference rollout covers 72 future frames at 10 Hz—7.2 seconds—and emits four-frame, 0.4-second blocks. NVIDIA says the separately trained two-step checkpoint is faster but sacrifices some visual quality relative to the four-step checkpoint. The company says the system runs interactively on one RTX PRO 6000 GPU; that is an NVIDIA performance claim, not an independently reproduced benchmark.

Real-time output does not make the simulator a biomechanical model. NVIDIA says visual realism is only a starting point for validation, not proof of biomechanical accuracy. Its published next steps include explicit geometry, contact and tissue mechanics, surgeon assessment and reproducible benchmarks. Those are research directions, not evidence of regulatory clearance, clinical benefit, suitability for training surgeons or safe control of a physical surgical robot.