Skip to content
Main Site News Console

# NVIDIA Cosmos3-H-Dreams: Realistically Recreating Clinical Surgeries via Generative Simulation

· Hugging Face Translated
教程模型卡

Imagine manipulating a Cadiere grasper to clamp and retract a gallbladder inside a real-time responsive, generative surgical scene. Today, we take the next step by officially releasing Cosmos3-H-Dreams, bringing interactive generative simulation to the visual and procedural complexities of clinical surgery with the most realistic in vivo imaging to date.

Our initial release, Cosmos-H-Dreams, built on Cosmos-Predict2.5, brought interactive generative simulation to da Vinci Research Kit (dVRK) tabletop suturing tasks—a foundational testbed in surgical robotics. As a World Foundation Model (WFM), Cosmos-H-Dreams demonstrated an alternative to traditional numerical simulation: rather than explicitly modeling scene geometry, tissue deformation, and physical interactions, it learns visual dynamics directly from synchronized video and robot kinematics data, enabling highly realistic interactive tabletop simulations.

Today, Cosmos3-H-Dreams advances this capability further by learning from videos and robot kinematics recorded during clinical surgeries performed on the CMR Surgical Versius platform. To our knowledge, this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports real-time kinematic control. Powered by Cosmos 3 and backed by NVIDIA’s accelerated streaming inference library FlashDreams, Cosmos3-H-Dreams runs interactively on just a single NVIDIA RTX PRO 6000 GPU.

Video 1


1. From Tabletop Tasks to Clinical Scenes

In controlled tabletop environments like the one used for Cosmos-H-Dreams, instruments operate against relatively simple backdrops. Inside the human body, however, even a slight tool movement can affect the entire scene: tissues deform and stretch under traction, specular highlights shift, anatomical structures become occluded, and surgical smoke obscures visibility. Given an initial image and input tool actions, a generative simulator must capture these dynamic responses while maintaining anatomical and instrumental consistency across each newly generated video chunk. Subtle errors can accumulate over time, gradually drifting…