08/01 release: Added support for spatial-temporal reasoning of city and industrial operations

README.md

@@ -17,7 +17,6 @@ tags:
 - cosmos
 ---
 
-
 # **Cosmos-Reason1: Physical AI Common Sense and Embodied Reasoning Models**
 
 [**Cosmos**](https://huggingface.co/collections/nvidia/cosmos-reason1-67c9e926206426008f1da1b7) | [**Code**](https://github.com/nvidia-cosmos/cosmos-reason1) | [**Paper**](https://arxiv.org/abs/2503.15558) | [**Paper Website**](https://research.nvidia.com/labs/dir/cosmos-reason1)
@@ -64,6 +63,7 @@ Physical AI: Space, time, fundamental physics understanding and embodied reasoni
 
 * Github: [05/17/2025](https://github.com/nvidia-cosmos/cosmos-reason1)
 * Huggingface:
+  * [08/01/2025](https://huggingface.co/nvidia/Cosmos-Reason1-7B/commit/1d4cbdc7a277affb4a69eca40b60b4479a5c63b8). Shipped a few improvements which include captions with temporal timestamp, Set of Mark prompting.
   * [06/10/2025](https://huggingface.co/nvidia/Cosmos-Reason1-7B/commit/2464fff43c5c0bfb1916ac8c009feda4aed81be9). Enhanced critic capability for physical plausibility.
   * [05/17/2025](https://huggingface.co/nvidia/Cosmos-Reason1-7B/commit/098a5bb62a1f4fc05e5c4ac89aae8005e301aa18). Initial release.
 
@@ -75,6 +75,21 @@ Network Architecture: Qwen2.5-VL-7B-Instruct.
 Cosmos-Reason-7B is post-trained based on [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) and follows the same model architecture.
 
 
+**Number of model parameters:**
+
+Cosmos-Reason1-7B:<br>
+  * Vision Transformer (ViT): 675.76M (675,759,104)
+  * Language Model (LLM): 7.07B (7,070,619,136)
+  * Other components (output projection layer): 545.00M (544,997,376)
+
+
+## Computational Load:
+
+  * Cumulative Compute: 3.2603016e+21 FLOPS
+  * Estimated Energy and Emissions for Model Training:
+    * Total kWh = 16658432
+    * Total Emissions (tCO2e) = 5380.674
+
 ## Input
 
   **Input Type(s)**: Text+Video/Image
@@ -95,7 +110,7 @@ Cosmos-Reason-7B is post-trained based on [Qwen2.5-VL-7B-Instruct](https://huggi
 
 ## Output
 
- **Output Type(s)**: Text 
+ **Output Type(s)**: Text
 
  **Output Format**: String
 
@@ -103,6 +118,9 @@ Cosmos-Reason-7B is post-trained based on [Qwen2.5-VL-7B-Instruct](https://huggi
 
  **Other Properties Related to Output**:
  * Recommend using 4096 or more output max tokens to avoid truncation of long chain-of-thought response.
+
+ * Our AI model recognizes timestamps added at the bottom of each frame for accurate temporal localization.
+
  * Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>
 
 
@@ -129,11 +147,47 @@ Cosmos-Reason-7B is post-trained based on [Qwen2.5-VL-7B-Instruct](https://huggi
 See [Cosmos-Reason1](https://github.com/nvidia-cosmos/cosmos-reason1) for details.
 * Post Training: [Cosmos-Reason1](https://github.com/nvidia-cosmos/cosmos-reason1) provides examples of supervised fine-tuning and reinforcement learning on embodied reasoning datasets.
 
-# Evaluation
-
+## Training and Evaluation Sections:
+### 05/17/2025
 Please see our [technical paper](https://arxiv.org/pdf/2503.15558) for detailed evaluations on physical common sense and embodied reasoning. Part of the evaluation datasets are released under [Cosmos-Reason1-Benchmark](https://huggingface.co/datasets/nvidia/Cosmos-Reason1-Benchmark). The embodied reasoning datasets and benchmarks focus on the following areas: robotics (RoboVQA, BridgeDataV2, Agibot, RobFail), ego-centric human demonstration (HoloAssist), and Autonomous Vehicle (AV) driving video data. The AV dataset is collected and annotated by NVIDIA.
+
 All datasets go through the data annotation process described in the technical paper to prepare training and evaluation data and annotations.
 
+### 08/01/2025
+We enhance the model capability with the augmented training data. PLM-Video-Human and Nexar are used to enable dense temporal captioning. Describe Anything is added to enhance a set of mark (SoM) prompting. We enrich data in intelligent transportation systems (ITS) and warehouse applications. Lastly, Visual Critics dataset contains a collection of AI generated videos from Cosmos-Predict2 and Wan2.1 with human annotations to describe the physical correctness in AI videos.
+
+
+## Training Datasets:
+
+**Data Collection Method**:
+* RoboVQA:  Hybrid:  Automatic/Sensors
+* BridgeDataV2: Automatic/Sensors
+* AgiBot: Automatic/Sensors
+* RoboFail: Automatic/Sensors
+* HoloAssist: Human
+* AV: Automatic/Sensors
+* PLM-Video-Human: Human
+* Nexar: Automatic/Sensors
+* Describe Anything: Human
+* ITS / Warehouse: Human, Automatic
+* Visual Critics: Automatic
+
+**Labeling Method**:
+* RoboVQA:  Hybrid:  Human,Automated
+* BridgeDataV2: Hybrid:  Human,Automated
+* AgiBot: Hybrid:  Human,Automated
+* RoboFail: Hybrid:  Human,Automated
+* HoloAssist: Hybrid:  Human,Automated
+* AV: Hybrid:  Human,Automated
+* PLM-Video-Human: Human,Automated
+* Nexar: Human
+* Describe Anything: Human,Automated
+* ITS / Warehouse: Human, Automated
+* Visual Critics: Human,Automated
+
+
+# Evaluation Datasets:
+
 **Data Collection Method**:
 * RoboVQA:  Hybrid:  Automatic/Sensors
 * BridgeDataV2: Automatic/Sensors
@@ -142,6 +196,7 @@ All datasets go through the data annotation process described in the technical p
 * HoloAssist: Human
 * AV: Automatic/Sensors
 
+
 **Labeling Method**:
 * RoboVQA:  Hybrid:  Human,Automated
 * BridgeDataV2: Hybrid:  Human,Automated
@@ -150,6 +205,7 @@ All datasets go through the data annotation process described in the technical p
 * HoloAssist: Hybrid:  Human,Automated
 * AV: Hybrid:  Human,Automated
 
+
 **Metrics**:
 We report the model accuracy on the embodied reasoning benchmark introduced in [Cosmos-Reason1](https://arxiv.org/abs/2503.15558). The results differ from those presented in Table 9 due to additional training aimed at supporting a broader range of Physical AI tasks beyond the benchmark.
 |                   | [RoboVQA](https://robovqa.github.io/)        | AV       | [BridgeDataV2](https://rail-berkeley.github.io/bridgedata/)| [Agibot](https://github.com/OpenDriveLab/AgiBot-World)| [HoloAssist](https://holoassist.github.io/)       | [RoboFail](https://robot-reflect.github.io/)                               | Average |
@@ -160,6 +216,7 @@ We report the model accuracy on the embodied reasoning benchmark introduced in [
 Modality: Video (mp4) and Text
 
 ## Dataset Quantification
+### 05/17/2025
 We release the embodied reasoning data and benchmarks. Each data sample is a pair of video and text. The text annotations include understanding and reasoning annotations described in the Cosmos-Reason1 paper. Each video may have multiple text annotations. The quantity of the video and text pairs is described in the table below.
 **The AV data is currently unavailable and will be uploaded soon!**
 
@@ -169,9 +226,13 @@ We release the embodied reasoning data and benchmarks. Each data sample is a pai
 | **RL Data**         | 252                                          | 200      | 240                                                  | 200                                            | 200                                            | N/A                                            | **2.6GB**           |
 | **Benchmark Data**  | 110                                          | 100      | 100                                                  | 100                                            | 100                                            | 100                                            | **1.5GB**           |
 
+We release text annotations for all embodied reasoning datasets and videos for RoboVQA and AV datasets. For other datasets, users may download the source videos from the original data source and find corresponding video sources via the video names. The held-out RoboFail benchmark is released for measuring the generalization capability.
 
+### 08/01/2025
+|                   | [PLM-Video-Human](https://huggingface.co/datasets/facebook/PLM-Video-Human) | Nexar       | [Describe Anything](https://huggingface.co/datasets/nvidia/describe-anything-dataset)| [ITS / Warehouse]       | Visual Critics                             | Total Storage Size |
+|------------------ |-----------------------------------------------------------------------------|-------------|--------------------------------------------------------------------------------------|-------------------------|--------------------------------------------|--------------------|
+| **SFT Data**      | 39k                                                                         | 240k        | 178k                                                                                 | 24k                     | 24k                                        | **2.6TB**          |
 
-We release text annotations for all embodied reasoning datasets and videos for RoboVQA and AV datasets. For other datasets, users may download the source videos from the original data source and find corresponding video sources via the video names. The held-out RoboFail benchmark is released for measuring the generalization capability.
 
 
 ## Inference:
@@ -310,4 +371,5 @@ We value you, the datasets, the diversity they represent, and what we have been
 | Model Application(s):                           | Physical AI common sense understanding and embodied reasoning                                                                                                                                                                                                                                                                                                                     |
 | Describe the life critical impact (if present). | None Known                                                                                                                                                                                                                                                                                                                           |
 | Use Case Restrictions:                          | [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license)                                                                                                                                                                                                                   |
-| Model and dataset restrictions:                 | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development.  Restrictions enforce dataset access during training, and dataset license constraints adhered to. Model checkpoints are made available on Hugging Face, and may become available on cloud providers' model catalog. |
+| Model and dataset restrictions:                 | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development.  Restrictions enforce dataset access during training, and dataset license constraints adhered to. Model checkpoints are made available on Hugging Face, and may become available on cloud providers' model catalog. |
+

model-00001-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cb3789b9843f08b8a181ee43fc074f79edfacb9081677b8f35fae69c34de9efd
+oid sha256:c28404126221997ae8eb70a23b919c96174d42e35ae1d537e0c95093d50b359a
 size 4968243304

model-00002-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:522dd37d8d1ed41b2134903d374167e7336c7f4e5eac3ca7b310dfbb42606a05
+oid sha256:f281081864c10992d3e03874c79d526c84407e049d713747f19eb9c79cd16db3
 size 4991495816

model-00003-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2f74ab185f8f51ceb0d5521dbe6ec51b11e16d986412269456a0e5c043f526a6
+oid sha256:3bb3a62a8d0e83c6283388ddea99395b221f908f9181b8edd0f7f91d02260ebe
 size 4932751040

model-00004-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5e0879e4d16019a1af04bec65f3058d114de947bd1412dd5cc8096b3fb7c6969
+oid sha256:91bacbe1ad798e16daa05023b4e4bec70b53c8cd7d757db86c5bc76c4e0bbf15
 size 1691924384