Generate Realistic Videos with NVIDIA COSMOS 1.0 Diffusion

NVIDIA Cosmos is a transformative platform that utilizes World Foundation Models (WFMs) to change the face of robotics training. The platform creates simulated environments in which robots can learn and adapt before real-world deployment by generating physically realistic videos. This article discusses the key components, risk mitigation strategies, and ethical considerations of using NVIDIA’s Cosmos-1.0-Diffusion models for generating physically aware videos.

Learning Objectives

• Get to know about NVIDIA’s Cosmos-1.0-Diffusion models.
• Explore the model’s key features and capabilities.
• Understand the architecture of NVIDIA’s Cosmos-1.0-Diffusion model in detail, including it’s various layers and embeddings.
• Learn the steps involved in downloading and setting up the model for generating physically realistic videos.

Introduction to NVIDIA’s Cosmos-1.0-Diffusion

The world of AI-generated content is constantly evolving, and NVIDIA’s Cosmos-1.0-Diffusion models are a giant leap forward in this area. This article dives into these powerful diffusion-based World Foundation Models (WFMs), which generate dynamic, high-quality videos based on text, images, or video inputs. Cosmos-1.0-Diffusion offers a suite of tools for developers and researchers to experiment with world generation and push the limits of what’s possible in AI-driven video creation.

It can be used to solve many Business Problems like:

• Warehouse Robot Navigation – Simulates optimal robotic paths to prevent congestion and improve efficiency.
• Predictive Maintenance – Generates clips of machine failure scenarios to detect early warning signs.
• Assembly Line Automation – Visualizes robotic workflows to refine processes before real deployment.
• Worker Training – Creates AI-driven training videos for safe machine operation and emergency handling.
• Quality Control – Simulates defect detection workflows to enhance AI-based inspection systems.

The Cosmos 1.0 release introduces several impressive models, each tailored for specific input types:

• Cosmos-1.0-Diffusion-7B/14B-Text2World: These models (7 billion and 14 billion parameters, respectively) generate 121-frame videos (approximately 5 seconds) directly from a text description. Imagine describing a bustling market scene, and the model brings it to life!
• Cosmos-1.0-Diffusion-7B/14B-Video2World: These models (also 7B and 14B parameters) take it a step further. Given a text description and an initial image frame, they predict the subsequent 120 frames, creating dynamic video continuations. This opens up exciting possibilities for video editing and content expansion.

Key Features and Capabilities

• High-Quality Video Generation: The models are designed to produce visually appealing videos with a resolution of 1280×704 pixels at 24 frames per second.
• Flexible Input: Cosmos-1.0-Diffusion supports text, image, and video inputs, providing developers with versatile tools for different use cases.
• Commercial Use Allowed: Released under the NVIDIA Open Model License, these models are ready for commercial applications, empowering businesses and creators to leverage this technology.
• Scalable Performance: NVIDIA provides guidance on optimizing inference time and GPU memory usage, allowing users to tailor performance to their hardware capabilities. They even offer model offloading strategies for GPUs with limited memory.

Model Architecture

The models use the diffusion transformer architecture with self-attention, cross-attention, and feedforward layers for denoising video in the latent space. It is possible for the model to condition on text input due to cross-attention, and the time information is embedded using adaptive layer normalization. Inputs of image or video are added by concatenating their latent frames with the generated frames.

The model follows a Transformer-based Diffusion Model approach for video denoising in latent space. Here’s a step-by-step breakdown:

Tokenization and Latent Space Processing

• The input video is first encoded using Cosmos-1.0-Tokenizer-CV8x8x8, converting it into a set of latent tokens.
• These tokens are then corrupted with Gaussian noise, making them partially degraded.
• A 3D Patchification step processes these tokens into non-overlapping 3D cubes, which serve as the input for the transformer network.

Transformer-Based Denoising Network

The model applies N blocks of:

• Self-Attention (for intra-frame and inter-frame relationships)
• Cross-Attention (to condition on text input)
• Feedforward MLP layers (to refine the denoising process)

Each block is modulated using adaptive layer normalization (AdaLN-LoRA), which helps stabilize training and improve efficiency.

a. Self-Attention (Understanding Spatiotemporal Relations)

• Self-attention is applied to the spatiotemporal latent tokens.
• It helps the model understand relationships between different video patches (both within frames and across frames).
• This ensures that objects and motion remain consistent across time.

b. Cross-Attention (Conditioning on Text Prompts)

• Cross-attention layers integrate the T5-XXL text embeddings as keys and values.
• This allows the model to align the generated video with the text description, ensuring semantic relevance.

c. Query-Key Normalization

• The paper mentions query-key normalization using RMSNorm.
• This helps prevent training instability where attention logits explode, ensuring smooth training.

d. MLP (Feedforward) Layers for Feature Refinement

• The MLP layers refine the denoised tokens.
• They apply additional transformations to improve clarity, texture details, and remove high-frequency noise.

Positional Embeddings for Temporal Awareness

The model uses 3D Rotary Position Embedding (3D RoPE) to embed positional information across:

• Temporal axis (time steps)
• Height axis (spatial dimension)
• Width axis (spatial dimension)

FPS-aware scaling is applied, ensuring the model generalizes to different frame rates.

Low-Rank Adaptation (AdaLN-LoRA)

• The model applies LoRA (Low-Rank Adaptation) to adaptive layer normalization (AdaLN).
• This significantly reduces model parameters (from 11B to 7B) while maintaining performance.

Final Reconstruction

• After N transformer layers, the denoised latent tokens are passed to the decoder of Cosmos-1.0-Tokenizer-CV8x8x8.
• The decoder converts the denoised tokens back into a video.

Input and Output

• Text2World Input: A text string (under 300 words) describing the desired scene, objects, actions, and background.
• Text2World Output: A 5-second MP4 video visualizing the text description.
• Video2World Input: A text string (under 300 words) and an image (or the first 9 frames of a video) with a resolution of 1280×704.
• Video2World Output: A 5-second MP4 video, using the provided image/video as a starting point and visualizing the text description for the subsequent frames.

Flow Diagram

How to Access Cosmos-1.0-Diffusion-7B-Text2World?

Now let’s learn how to access NVIDIA’s Cosmos-1.0-Diffusion-7B-Text2World model and set it up for generating physically realistic videos.

1. Setup

Install Libraries

pip install requests streamlit python-dotenv

2. Download the Model

There are 2 ways to download the model – either through Hugging Face or through the API.

Hugging Face: Download the model from here.

Through API Key: To use Cosmos-1.0 Diffusion Model through API Key we need to checkout NVIDIA NIM.

3. Store API key in .env File

NVIDIA_API_KEY="Your_API_KEY"

How to Generate Physically Realistic Videos Using Cosmos-1.0-Diffusion-7B-Text2World?

Now that we’re all

1. Importing Required Libraries

import requests
import streamlit as st
from dotenv import load_dotenv
import os

2. Setting Up API URLs and Loading Environment Variables

invoke_url = "https://ai.api.nvidia.com/v1/cosmos/nvidia/cosmos-1.0-7b-diffusion-text2world"
fetch_url_format = "https://api.nvcf.nvidia.com/v2/nvcf/pexec/status/"
load_dotenv()
api_key = os.getenv("NVIDIA_API_KEY")

• invoke_url: The endpoint to send prompts and generate AI-driven videos.
• fetch_url_format: Used to check the status of the request using a unique request ID.
• load_dotenv(): Loads environment variables from a .env file.

3. Setting Up API Headers

headers = {
    "Authorization": f"Bearer {api_key}",
    "Accept": "application/json",
}

4. Creating the Streamlit UI

st.title("NVIDIA Text2World")
prompt = st.text_area("Enter your prompt:", "A first person view from the perspective from a human sized robot as it works in a chemical plant. The robot has many boxes and supplies nearby on the industrial shelves. The camera on moving forward, at a height of 1m above the floor. Photorealistic")

5. Handling User Input and API Request Execution

if st.button("Generate"):

• Waits for the user to click the “Generate” button before executing the API request.

6. Preparing the API Request Payload

payload = {
    "inputs": [
        {
            "name": "command",
            "shape": [1],
            "datatype": "BYTES",
            "data": [
                f"text2world --prompt=\"{prompt}\""
            ]
        }
    ],
    "outputs": [
        {
            "name": "status",
            "datatype": "BYTES",
            "shape": [1]
        }
    ]
}

• inputs: Specifies the command format for NVIDIA’s Text2World model, embedding the user’s prompt.
• outputs: Requests the status of the AI-generated video.

7. Sending the API Request and Handling the Response

session = requests.Session()
response = session.post(invoke_url, headers=headers, json=payload)

• requests.Session(): Reuses connections for efficiency.
• session.post(): Sends a POST request to initiate the AI video generation.

8. Polling Until the Request Completes

while response.status_code == 202:
    request_id = response.headers.get("NVCF-REQID")
    fetch_url = fetch_url_format + request_id
    response = session.get(fetch_url, headers=headers)

• Checks if the request is still in progress (202 status code).
• Extracts the unique NVCF-REQID from headers to track request status.
• Continuously sends GET requests to fetch the updated status.

9. Handling Errors and Saving the Result

response.raise_for_status()
with open('result.zip', 'wb') as f:
    f.write(response.content)

• raise_for_status(): Ensures any request failure is properly reported.
• Writes the generated video data into a result.zip file.

10. Notifying the User of Completion

st.success("Generation complete! Check the result.zip file.")

• Displays a success message once the file is saved.

Get Code from GitHub Here

Output

Now let’s try out the model:

Prompt

“A first-person view from the perspective of a life-sized humanoid robot as it operates in a chemical plant. The robot is surrounded by numerous boxes and supplies neatly arranged on industrial shelves. The camera moves forward at a height of 1 meter above the floor, capturing a photorealistic scene.”

Video Output

Conclusion

This project shows how NVIDIA’s Text2World can create AI-driven, physically realistic videos based on textual prompts. We built an intuitive interface where users are able to visualize AI-generated environments efficiently with the use of Streamlit for user interaction as well as requests for API communication. The system continuously monitors the status of the requests and thus ensures smooth running and retrieval of the generated content.

Such AI models have vast applications in robotics simulation, industrial automation, gaming, and virtual training, enabling realistic scenario generation without the need for expensive real-world setups. As generative AI evolves, it will further bridge the gap between virtual and real-world applications, enhancing efficiency and innovation across various industries.

Key Takeaways

• NVIDIA’s Cosmos-1.0-Diffusion generates high-quality, physics-aware videos from text, images, or videos, making it a key tool for AI-driven world simulation.
• The model accepts text descriptions (Text2World) and text + image/video (Video2World) to create realistic 5-second videos at 1280×704 resolution, 24 FPS.
• Cosmos runs on NVIDIA GPUs (Blackwell, Hopper, Ampere), with offloading strategies available for memory-efficient execution, requiring 24GB+ GPU memory for smooth inference.
• Released under the NVIDIA Open Model License, Cosmos allows for commercial use and derivative model development, making it ideal for industries like robotics, gaming, and virtual training.
• NVIDIA emphasizes Trustworthy AI by implementing safety guardrails and ethical AI practices, ensuring responsible usage and preventing misuse of generated content.

Frequently Asked Questions

Gourav Lohar

Hi I'm Gourav, a Data Science Enthusiast with a medium foundation in statistical analysis, machine learning, and data visualization. My journey into the world of data began with a curiosity to unravel insights from datasets.