Nvidia made four Cosmos 3 Nano open-weight checkpoints available on Hugging Face on 5 November, extending its Cosmos platform for simulated driving and robotics with models that generate visual observations from scene controls or actions. In its release post on Hugging Face, Nvidia describes the checkpoints as post-trained variants of Cosmos 3 Nano, each with 16B trainable parameters. Developers can download the weights and use published post-training recipes to adapt the models to their own data and controls.
Key points
- Two checkpoints generate driving observations, one for offline data and one for streaming simulation.
- Two others generate video from simulator controls or two-arm robot actions.
- The weights carry the OpenMDW1.1 licence, which permits commercial and non-commercial use under its terms.
- Nvidia says the models’ usefulness for training and evaluating policies still needs to be measured.
Four Cosmos 3 Nano routes into simulation
The checkpoints are Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer and Cosmos3-Nano-Sim-Bimanual. Together they cover two different uses of generated video: producing recorded scenes for later work and producing the next observation as an interactive simulation proceeds. Nvidia says the open weights and post-training recipes are intended as a starting point for adaptation and comparison. The models use the OpenMDW1.1 licence, permitting commercial and non-commercial use under its terms.
Nvidia’s 2 November developer guide described the simulation checkpoints as available through Cosmos Framework, its collection of training, optimisation and inference tools. The 5 November release presents them as newly available on Hugging Face. AI Affairs previously reported Nvidia’s wider open-model range for robotics, biology and quantum computing.
Transfer-Auto and Sim-Auto divide the driving work
Cosmos3-Nano-Transfer-Auto generates an entire clip for offline driving data. Nvidia says it can keep video from up to 11 camera views in sync and jointly generate colour images and LiDAR observations, the distance-based view used for three-dimensional sensing. Maps supplied separately for each camera control road structure and traffic movement, while text prompts guide appearance. That combination is meant to let a developer vary conditions such as weather or lighting while retaining the scene’s layout.
Driving in rain or darkness could therefore be represented as different-looking versions of a controlled road scene, rather than requiring a different layout for each condition. If those generated views remain aligned with their distance observations, they could provide varied scenes in which to train or evaluate driving behaviour.
Cosmos3-Nano-Sim-Auto takes a different route: it generates successive short portions of camera video and LiDAR observations using earlier output and updated driving controls. For closed-loop driving, where an action changes the scene before the next observation arrives, Nvidia says it also needs a simulation runtime and a renderer to update those controls. Developers adapting the system to another sensor arrangement or driving domain are directed to post-train Transfer-Auto first and then distil it into the streaming variant.
Nvidia says its published Transfer-Auto reference run, using a sample configuration with seven synchronised camera views, produces seven 720p videos of 297 frames at 30 FPS and 99 LiDAR sweeps. Its Sim-Auto reference run instead uses pre-rendered controls to generate observations successively. Those are described runs with prepared inputs; Nvidia says measuring how faithfully the models support policy learning and evaluation, and finding where they fall short, remains work to be done.
Sim-Transfer and Sim-Bimanual generate robot views
Cosmos3-Nano-Sim-Transfer turns controls from a physics simulator into colour video guided by a text prompt. The controls can describe depth, object regions, scene edges or coarse appearance. The simulator remains responsible for motion, contacts and physical state; the model supplies the resulting visual observation. Nvidia describes this as a way to change the look of a simulated scene while keeping its underlying motion with the simulator.
Cosmos3-Nano-Sim-Bimanual instead predicts the next visual observation from an image and the actions of a two-arm robot or camera. Nvidia says it supports YAM and AgiBot embodiments, with control supplied by a policy or human-operated joysticks. Its action format specifies coordinate frames, rotations and gripper calibration, among other conventions, so the values sent to the model describe the intended movements.
For an interactive sequence, the distinction matters: Sim-Transfer receives a state already updated by a physics simulator, whereas Sim-Bimanual generates an observation directly from the scene and the supplied actions. Nvidia says Sim-Bimanual’s training data must align each run of action transitions with the observations on either side of them.
Cosmos 3 predicts video in short chunks
Cosmos 3 uses a Mixture-of-Transformers design that connects separate networks for processing text and generating visuals through attention. For streaming, the generator works a short piece at a time, drawing on the controls and video it has already produced. It is rather like drawing the next frame of a flipbook while looking at the preceding pages: a new instruction can affect the next piece, but the generator has to work with the history it has been given.
Nvidia describes a four-stage training recipe, beginning with adaptation to complete recorded clips, then training on earlier recorded observations while hiding the future answer. A frozen teacher model subsequently guides a student to produce each piece in four denoising steps. The final stage trains that student using its own generated history, addressing the problem that an inaccurate object in one piece can become part of the context for later pieces.
The Sim-Bimanual instructions call for Linux, an Nvidia GPU, Docker and Nvidia Container Toolkit. Nvidia says its published vLLM-Omni setup was tested on one GB200 GPU and that the test is not a minimum-memory specification.
