Physical AI: 机器人领域的 1000 亿美元新蓝海

Utkarsh 2026-04-28 45.9 KB 184.5K 284 𝕏 文章列表

文章深入剖析了 NVIDIA 在机器人领域构建的隐秘且强大的操作系统，这是一个由 Cosmos（想象力/仿真）、GR00T（大脑/推理）和 Newton（物理引擎/身体）组成的全栈技术体系。文章指出，虽然大众关注机器人的硬件动态，但 NVIDIA 通过开源 Cosmos Cookbook 等工具，已经为开发者解决了物理世界 AI 中数据稀缺和仿真模拟等核心难题，标志着机器人开发范式的根本性转变。

# Physical AI: The $100B Layer You’re Not Building On **作者**: Utkarsh **日期**: 2022-01-19T17:58:45.000Z **来源**: [https://x.com/utk7arsh/status/2048812835514298739](https://x.com/utk7arsh/status/2048812835514298739) --- ![Image](/doc/1228/image/img_50e3373e19b7.jpg) Everyone is watching robots walk. Boston Dynamics. Figure AI. Tesla Optimus. The demos are everywhere. The hype is everywhere. Nobody is watching what the robots are thinking. And while the world was busy watching the videos, NVIDIA quietly dropped the entire operating system for physical AI. Open source but not tapped by most developers. That window is closing. This is everything you need to know, and exactly how to get started today. ## The Stack Nobody Explained Here is the story most people know: > NVIDIA makes chips. GPUs go brrr. Blackwell. Trillion dollar company. Here is the story most people missed: Over the last 18 months, NVIDIA assembled the complete operating system for the physical world. Not a chip. Not a single model. A full 3-piece stack that each solve a different impossible problem in robotics, and together represent the most important infrastructure shift since the transformer architecture. The three pieces are Cosmos, GR00T, and Newton. Each one is massive on its own. Together, they are a paradigm shift. Piece 1: Cosmos: The Robot's Imagination The biggest bottleneck in robotics has never been the hardware. It has been data. To train a robot to do anything useful, like pick up a glass, navigate a warehouse, or sort packages, you need thousands of hours of real-world video showing that task in every possible condition. Different lighting. Different surfaces. Different angles. Different environments. That data collection used to take months. Sometimes years. And even then it was never diverse enough. Cosmos solves this at the foundation level. Cosmos is a world foundation model — a neural network trained on millions of hours of real driving and robotics video that can generate physics-aware simulations of the future. You give it a simulation. It outputs 1,000 photorealistic variations with different lighting, different materials, different weather, and camera angles without you running a single real-world trial. One simulation becomes a thousand training samples in hours that earlier took months. But that is just Cosmos Predict. The full Cosmos platform has three sub-models: Cosmos Predict 2.5: generates future-state video predictions from text, image, or existing video input. You describe a scenario. It simulates it. Tapping the Physics accuracy. Cosmos Transfer 2.5: takes synthetic simulation footage and transforms it into photorealistic scenes. A robot arm moving in a sterile simulation lab becomes the same robot arm moving in a real kitchen with worn countertops and afternoon shadows. The geometry and motion are preserved. The visual domain is completely changed. This is the sim-to-real bridge that has been the holy grail of robotics for a decade. Cosmos Reason 2: a 7B reasoning vision-language model that can watch video and think about what it sees. Not just classify. Not just detect. Reason. Plan. Predict. Piece 2: GR00T: The Robot's Brain In March 2025, NVIDIA released something that should have broken the internet. Isaac GR00T N1: the world's first open, fully customizable foundation model designed specifically for humanoid robots It is on Hugging Face. Right now. Free to download and fine-tune. This is the GPT moment for robotics or a predecessor to that moment in my opinion. What GR00T actually does is give a robot generalized reasoning and motor skill capabilities before you teach it anything domain-specific. Instead of building a robot brain from scratch for every application, you start from GR00T and fine-tune. Think about what that means. The same way, developers don't train a language model from scratch to build a chatbot. They fine-tune on top of an existing foundation. Robotics developers can now fine-tune on top of GR00T instead of starting from zero every single time. Companies like AeiROBOT, Franka Robotics, LG Electronics, Lightwheel, Mentee Robotics, Neura Robotics, Solomon, and UCR are already evaluating it. And it is open. You can start today. Piece 3: Newton: The Robot's Body Physics This is the piece that almost nobody is talking about, and it might be the most important. Before a robot learns anything in the real world, it learns in simulation. The quality of that simulation, especially how accurately it models gravity, friction, contact, mass, inertia, and the way a cardboard box bends when you grip it, directly determines how well the robot performs when it leaves the virtual environment and enters your warehouse. To understand more of how this works, check out Isaaclab or ManiSkill3 environments in which engines like Newton can be integrated. Garbage physics in simulation = a robot that breaks things in the real world. NVIDIA built Newton to fix this. It is a new open-source physics engine purpose-built for robotics. GPU-accelerated, running CUDA-level speed without requiring low-level coding that cuts simulation times from days to minutes. Newton was built in collaboration with Google DeepMind and Disney Research. Disney. The company that has been simulating physics for animated films and theme park systems for 30 years. The people who figured out how water moves, how hair falls, how cloth drapes. That same team is now teaching robots how to manipulate objects with their hands. NVIDIA's VP of Omniverse and simulation technology put the whole picture together in one sentence: ![Image](/doc/1228/image/img_b2346cc00837.jpg) Pic credits: Nvidia newsroom ## The GitHub Repo That Changes Everything Now here is where it gets real. NVIDIA did not just release models and documentation. They released a cookbook. A GitHub repo called the Cosmos Cookbook, containing step-by-step workflows, technical recipes, and complete runnable examples for building, adapting, and deploying world foundation models. This is not a research paper. This is working code, organized like a recipe book, built for developers who want to build something today and not in two years. The repo: https://github.com/nvidia-cosmos/cosmos-cookbook The documentation site: https://nvidia-cosmos.github.io/cosmos-cookbook You clone it in 30 seconds: ``` git clone https://github.com/nvidia-cosmos/cosmos-cookbook.git cd cosmos-cookbook git lfs install # Required — the repo contains video files ``` ## What Is Actually Inside Let me walk you through what is in the cookbook right now, because the recipes are wild. Recipe 1: Egocentric Social and Physical Reasoning The recipe uses Cosmos Reason 2 from a first-person robot camera POV. You feed it a video of a human doing something in front of the robot, and you ask it to interpret the scene and predict what happens next. In a live demo from NVIDIA's own cookbook, researchers showed a person initiating a fist bump at the robot's camera and asked: "Is this person doing a fist bump at me? Can you predict what is the most likely position her fist will be in the next 2 seconds?" Cosmos Reason 2 answered: Yes, the person is fist-bumping. Her fist will likely remain in a fistbump position, maintaining contact with the viewer's fist as she holds the gesture. A competing model thought it was a pointing gesture. A commanding, assertive motion. Not an interaction. That gap between a robot that understands intent and a robot that just sees motion is the entire game in human-robot interaction. Cosmos Reason 2 gets it right. And it is 8 billion parameters. It runs on a single GPU. Recipe 2: Sim-to-Real Transfer for Robotic Navigation Task The cookbook includes a full walkthrough of taking synthetic robot simulation footage of a robot arm working in a sterile, fake kitchen environment. Using Cosmos Transfer 2.5, it is able to generate photorealistic variations with different cabinet styles, surface materials, lighting conditions, and kitchen layouts. The robot's geometry and motion paths are preserved exactly. But the visual domain changes completely. You end up with training data that looks real, without any real-world data collection. Robotics engineers can now generate diverse training data from a single simulation. One simulation run becomes ten visual domains. One environment becomes fifty. The data flywheel that used to require field operations now runs in the cloud overnight. Recipe 3: Physical Plausibility Prediction This one is quieter but just as important. Synthetic data generation is a cheat code until you accidentally train your robot on physically impossible scenarios. If your synthetic video shows a cup sliding through a table instead of bouncing off it, your robot learns the wrong physics. This recipe is a quality filter. Cosmos Reason 2 evaluates generated video frames for physical plausibility before they enter the training pipeline. It flags footage where the physics are wrong and removes it automatically. It is a QA layer for reality itself. And it runs before your training loop ever sees the data. ## What People Are Already Building Real deployments are running right now. Agriculture: A company called Aigen is training autonomous rovers to distinguish crops from weeds across millions of simulated farm scenarios using Cosmos and Isaac Sim. Every field is different because of different crops, different soil, different growth stages. Cosmos generates the variety across all those scenarios in simulation. Each rover then deploys on an NVIDIA Jetson Orin edge AI module and makes real-time weed detection decisions in the field. The thing that would have required 10,000 hours of field data collection now happens in a simulation pipeline. ![Image](/doc/1228/image/img_5dde393efdd7.jpg) Pic credits:aigen.io > **TechCrunch@TechCrunch**: [原文链接](https://x.com/TechCrunch/status/1483861501765165057) > > Aigen's swarm of agtech robots want to make agriculture carbon negative > https://techcrunch.com/2022/01/18/aigen-fund-round/… by @Haje Surgical Robotics: One of the newer recipes fine-tunes Cosmos Predict 2.5 as an action-conditioned surgical simulator. The result, Cosmos-H-Surgical-Simulator, is a collaboration between NVIDIA, Johns Hopkins, and 35 institutions. The efficiency gain alone tells the story: 600 policy rollouts in 40 minutes versus 2 days of real-world benchtop testing. A prototype of GR00T-H has already completed a full end-to-end suture on the SutureBot benchmark. Sources: Hugging Face · Open-H-Embodiment · GitHub > **NVIDIA Healthcare@NVIDIAHealth**: [原文链接](https://x.com/NVIDIAHealth/status/2034038372495167495) > > Imagine a robot that can suture tissue as skillfully as a surgeon - turning AI from perception into precise, life‑saving action. > Explore the three breakthroughs powering this leap: > Open‑H‑Embodiment - the first open, community‑driven dataset delivering standardized > > ![图片](/doc/1228/image/img_68145dc4e026.jpg) Industrial Safety: The Cookoff competition drew over 1,600 participants. The first-place winner, Zenith, built an explainable visual reasoning system for mixed palletizing, where a robot reasons out loud about how it's sorting packages. Second place built ResQ-AI, an autonomous disaster response drone. Source: NVIDIA Developer Forums — Cookoff Winners ## How a Complete Beginner Gets Started Today You do not need a robot. You do not need a warehouse. You do not need a PhD in robotics. You need a GPU and a Hugging Face account. If you do not have a local GPU, you can spin up a cloud instance on Nebius or Brev. Both have ready-to-launch Cosmos environments. Here is the exact path. Step 1: Set Up the Environment (discussed before) before cloning.If you are on Mac or Windows, check the official Git LFS installation guide atgit-lfs.com For cloud deployment, the cookbook documentation lists ready-to-go GPU instances on Nebius and Brev that come pre-configured. You can skip the local setup entirely. Step 2: Run Your First Cosmos Reason 2 Inference This is the fastest way to feel what the stack can actually do. The Cosmos Reason 2 model comes in two sizes: 2B parameters and 8B parameters. The 2B model runs on a consumer GPU. Both are on Hugging Face. ``` from transformers import AutoProcessor, AutoModelForVision2Seq import torch # Load the 2B model — runs on a single consumer GPU model_id = "nvidia/Cosmos-Reason2-2B" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForVision2Seq.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto" ) # Ask it a physics question about a video prompt = """Watch this video carefully. Think step by step about what is physically happening. <think> your reasoning </think> <answer> your answer </answer> Question: What action is being performed and what will happen next?""" ``` The chain-of-thought output (the <think> block) is where the magic is. Watch it reason about physics, spatial relationships, and likely outcomes. That is not a classifier. That is a model building a world model in real time. Step 3: Try the Egocentric Reasoning Recipe This is the most immediately mind-bending entry point. Record a short video from a first-person perspective (even works from your phone camera) of someone doing something with their hands. Handing you an object. Waving. Picking something up. Feed it to Cosmos Reason 2 and ask: "What is this person doing? What are they about to do? What should I do in response?" The model will chain-of-thought through its answer invoking spatial understanding, temporal reasoning, and physical sense. This will not only give you not just a description but a prediction and a recommended action. That is the brain of a robot. And you just ran it. The full recipe is documented here: https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html Step 4: Generate Your First Synthetic Training Data If you have access to Isaac Sim, or even just a simple 3D simulation environment, you can use Cosmos Transfer 2.5 to generate domain-adapted training data. The workflow: 1. Run a simulation of any robotic task. Even something simple like a robotic arm reaching for an object works. 2. Export the video and depth maps from your simulation 3. Feed into Cosmos Transfer 2.5 with edge control enabled 4. Specify a target visual domain — "industrial warehouse, fluorescent lighting, concrete floor" or "home kitchen, morning light, granite countertop" Cosmos Transfer preserves the robot's geometry and motion exactly. It changes the visual environment completely. You now have training data that looks like it was captured in a real deployment environment. The sim-to-real gallery in the cookbook shows the output quality visually: https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html Step 5: Go Deeper and build the end-to-end Recipe When you are ready to go from individual recipes to a full pipeline, the cookbook includes an end-to-end recipe called GR00T Dreams that chains Cosmos Predict 2.5 and Cosmos Reason 2 together. Setup guide: https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html This is the recipe that takes you from "I understand what this stack does" to "I am building a physical AI pipeline." It requires Ampere-architecture NVIDIA GPUs (RTX Pro 6000, A100, H100) or newer, with CUDA 12.8+. If you do not have that locally, use the cloud instances. ## What You Should Build Here are five concrete project ideas you can start this week without any robotics background. 1. A workplace safety monitor: Use the VSS blueprint to analyze camera footage from any environment — your office, a construction site, a kitchen — and have Cosmos Reason 2 describe what is happening, flag potential hazards, and generate a daily safety report. Zero labeled data. Zero custom training. Just prompting. 2. A sim-to-real data augmentation pipeline: Take any 3D environment you can simulate (even a simple Blender scene) and run it through Cosmos Transfer to generate photorealistic variants. Build a dataset generator that produces visual diversity on demand. 3. A physical plausibility filter: Build a data curation tool that takes generated video from any source (not just Cosmos) and runs it through Cosmos Reason 2 to check whether the physics make sense. Pipe it into any robotics training workflow as a quality gate. 4. An egocentric reasoning API: Wrap Cosmos Reason 2 in a simple FastAPI endpoint. Accept video uploads. Return chain-of-thought reasoning about what is happening and what should happen next. Sell it as an inference endpoint to robotics teams that do not want to host the model themselves. 5. An agricultural monitoring proof of concept: Point a camera at any outdoor environment with plants and objects. Ask Cosmos Reason 2 to identify, classify, and track objects over time. The Aigen story shows this scales to industrial farming. It starts with a camera and a prompt. ## Why Right Now Is The Moment Let me be direct about the timing. The Cosmos Cookbook launched publicly in late 2025. The Cookoff hackathon ran January through March of this year. Cosmos Reason 2 dropped in December 2025. GR00T N1.5 followed. Newton 1.0 just went general availability last week. This stack is weeks old in its current form. The community is tiny. The number of people who have actually cloned this repo and run the recipes is a rounding error compared to the number of people who will be using it in 18 months. In software, the ChatGPT moment created the LLM developer wave in early 2023. The people who built on top of GPT-3.5 in January 2023 were a year ahead of the market. Most developers caught it late. This is the same inflection. Except this time the entire stack is open including the models, the physics engine, the simulation framework, the cookbook. NVIDIA is not gating access. They are actively trying to democratize it. There is a version of this moment where you are the person who figured it out early. That version requires cloning the repo today. ## The Resources, All In One Place Main Cookbook GitHub: https://github.com/nvidia-cosmos/cosmos-cookbook Full Cosmos GitHub Organization (all repos): https://github.com/nvidia-cosmos Cosmos Reason 2 GitHub: https://github.com/nvidia-cosmos/cosmos-reason2 Cosmos Cookbook Documentation Site: https://nvidia-cosmos.github.io/cosmos-cookbook Cosmos Reason 2 on Hugging Face (2B model): https://huggingface.co/nvidia/Cosmos-Reason2-2B Cosmos Reason 2 on Hugging Face (8B model): https://huggingface.co/nvidia/Cosmos-Reason1-7B (search Cosmos Reason 2) Try Cosmos Reason 2 Without Any Setup: https://build.nvidia.com Egocentric Reasoning Recipe: https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html Sim-to-Real Robotics Gallery: https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html GR00T Dreams End-to-End Recipe: https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html Newton Physics Engine: https://developer.nvidia.com/newton-physics NVIDIA Cosmos Platform Overview: https://www.nvidia.com/en-us/ai/cosmos/ The ChatGPT moment for robotics is happening right now. The entire stack is open. Go build something. ## 相关链接 - [Utkarsh](https://x.com/utk7arsh) - [@utk7arsh](https://x.com/utk7arsh) - [63K](https://x.com/utk7arsh/status/2048812835514298739/analytics) - [https://github.com/nvidia-cosmos/cosmos-cookbook](https://github.com/nvidia-cosmos/cosmos-cookbook) - [https://nvidia-cosmos.github.io/cosmos-cookbook](https://nvidia-cosmos.github.io/cosmos-cookbook) - [Recipe 1](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html) - [Recipe 2](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/transfer1/inference-x-mobility/inference.html) - [Recipe 3](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/post_training/reason2/physical-plausibility-check/post_training.html) - [aigen.io](https://aigen.io/) - [Jan 20, 2022](https://x.com/TechCrunch/status/1483861501765165057) - [@Haje](https://x.com/Haje) - [Hugging Face](https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator) - [Open-H-Embodiment](https://huggingface.co/blog/nvidia/physical-ai-for-healthcare-robotics) - [GitHub](https://github.com/NVIDIA-Medtech/Cosmos-H-Surgical) - [Mar 18](https://x.com/NVIDIAHealth/status/2034038372495167495) - [7.4K](https://x.com/NVIDIAHealth/status/2034038372495167495/analytics) - [NVIDIA Developer Forums — Cookoff Winners](https://forums.developer.nvidia.com/t/the-results-are-in-meet-the-nvidia-cosmos-cookoff-winners-see-them-live-on-april-16/366130) - [git-lfs.com](https://git-lfs.com/) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html) - [https://github.com/nvidia-cosmos/cosmos-cookbook](https://github.com/nvidia-cosmos/cosmos-cookbook) - [https://github.com/nvidia-cosmos](https://github.com/nvidia-cosmos) - [https://github.com/nvidia-cosmos/cosmos-reason2](https://github.com/nvidia-cosmos/cosmos-reason2) - [https://nvidia-cosmos.github.io/cosmos-cookbook](https://nvidia-cosmos.github.io/cosmos-cookbook) - [https://huggingface.co/nvidia/Cosmos-Reason2-2B](https://huggingface.co/nvidia/Cosmos-Reason2-2B) - [https://huggingface.co/nvidia/Cosmos-Reason1-7B](https://huggingface.co/nvidia/Cosmos-Reason1-7B) - [https://build.nvidia.com](https://build.nvidia.com/) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html) - [https://developer.nvidia.com/newton-physics](https://developer.nvidia.com/newton-physics) - [https://www.nvidia.com/en-us/ai/cosmos/](https://www.nvidia.com/en-us/ai/cosmos/) - [Upgrade to Premium](https://x.com/i/premium_sign_up) - [1:13 AM · Apr 28, 2026](https://x.com/utk7arsh/status/2048812835514298739) - [63.4K Views](https://x.com/utk7arsh/status/2048812835514298739/analytics) - [View quotes](https://x.com/utk7arsh/status/2048812835514298739/quotes) --- *导出时间: 2026/4/28 17:41:24* --- ## 中文翻译 # 物理智能（Physical AI）：那层价值千亿美元、你却还没在上面构建的基础设施 **作者**： Utkarsh **日期**： 2022-01-19T17:58:45.000Z **来源**： [https://x.com/utk7arsh/status/2048812835514298739](https://x.com/utk7arsh/status/2048812835514298739) --- ![Image](/doc/1228/image/img_50e3373e19b7.jpg) 所有人都在看机器人走路。波士顿动力。Figure AI。特斯拉 Optimus。演示视频到处都是，炒作也无处不在。却没有人关注机器人在“想”什么。就在全世界忙着看视频的时候，NVIDIA 悄然推出了完整的物理智能操作系统。它是开源的，但大多数开发者并未加以利用。这个窗口期正在关闭。这就是你需要了解的一切，以及今天就能上手的正确方法。 ## 没有人解释过的技术栈大多数人都知道这样一个故事： > NVIDIA 造芯片。GPU 疯狂运转。Blackwell。万亿美元市值的公司。而大多数人错过的是这个故事：在过去的 18 个月里，NVIDIA 组装出了完整的物理世界操作系统。不是芯片，也不是单一模型，而是一个由三部分组成的完整技术栈，每一部分都解决了机器人技术中一个不同的“不可能难题”，合在一起则构成了自 Transformer 架构以来最重要的基础设施变革。这三块是 Cosmos、GR00T 和 Newton。每一块单独拿出来都分量十足。合在一起，它们是一场范式转变。第一块：Cosmos：机器人的想象力机器人技术最大的瓶颈从来不是硬件，而是数据。要训练机器人做任何有用的事情，比如拿起一个杯子、在仓库中导航或分拣包裹，你需要数千小时的真实世界视频，展示该任务在各种可能条件下的表现。不同的光线。不同的表面。不同的角度。不同的环境。这种数据采集过去需要几个月，有时甚至几年。即便如此，多样性也永远不够。 Cosmos 在基础层面解决了这个问题。 Cosmos 是一个世界基础模型——一个在数百万小时真实驾驶和机器人视频上训练过的神经网络，能够生成具有物理感知能力的未来模拟。你给它一个模拟场景，它就能输出 1,000 个逼真的变体，光线不同、材质不同、天气不同、相机角度不同，而你无需进行任何一次真实世界的试验。一个模拟场景在几小时内变成一千个训练样本，而这在以前需要数月。但这只是 Cosmos Predict。完整的 Cosmos 平台包含三个子模型： Cosmos Predict 2.5：从文本、图像或现有视频输入生成未来状态的视频预测。你描述一个场景，它就模拟出来，并保持物理层面的准确性。 Cosmos Transfer 2.5：将合成的模拟画面转换为照片级逼真的场景。在无菌模拟实验室里移动的机械臂，变成了在真实厨房里移动的同一条机械臂，台面有磨损，午后有阴影。几何结构和运动得以保留，视觉域则完全改变。这就是 sim-to-real（仿真到现实）的桥梁，十年来机器人技术一直追求的圣杯。 Cosmos Reason 2：一个 7B 参数的推理型视觉-语言模型，可以观看视频并思考它所看到的内容。不只是分类，不只是检测，而是推理、规划、预测。第二块：GR00T：机器人的大脑 2025 年 3 月，NVIDIA 发布了一个本应引爆互联网的东西。 Isaac GR00T N1：世界上首个开放、完全可定制、专为人形机器人设计的基础模型。它就在 Hugging Face 上，现在就可以免费下载和微调。在我看来，这就是机器人领域的 GPT 时刻，或者说那个时刻的先声。 GR00T 真正做的事，是让你在教机器人任何领域专属技能之前，先赋予它通用推理和运动能力。你不必再为每个应用从零开始构建机器人大脑，而是从 GR00T 出发进行微调。想想这意味着什么。就像开发者构建聊天机器人时不会从零训练一个语言模型，而是在现有基础模型上微调一样。机器人开发者现在可以在 GR00T 之上进行微调，而不必每次都从零开始。 AeiROBOT、Franka Robotics、LG 电子、Lightwheel、Mentee Robotics、Neura Robotics、Solomon 和 UCR 等公司已经在评估它。而且它是开放的。你今天就可以开始。第三块：Newton：机器人的身体物理这是几乎没有人谈论的一块，而它可能是最重要的。机器人在真实世界学习任何东西之前，都先在模拟中学习。那次模拟的质量，尤其是它对重力、摩擦、接触、质量、惯性以及你抓握时纸箱如何弯折这些东西的建模精确度，直接决定了机器人离开虚拟环境、进入你的仓库时的实际表现。想进一步了解其工作原理，可以看看 Isaaclab 或 ManiSkill3 环境，Newton 这类引擎可以集成其中。模拟中的垃圾物理 = 真实世界中会弄坏东西的机器人。 NVIDIA 打造 Newton 就是为了解决这个问题。它是一个专为机器人技术而生的全新开源物理引擎。GPU 加速，以 CUDA 级别的速度运行，无需底层编码，将模拟时间从几天缩短到几分钟。 Newton 是与 Google DeepMind 和迪士尼研究院（Disney Research）合作开发的。迪士尼。那家 30 年来一直在为动画电影和主题公园系统模拟物理的公司。那群弄清楚了水如何流动、头发如何垂落、布料如何垂坠的人。如今，同样的团队正在教机器人如何用手操控物体。 NVIDIA Omniverse 与模拟技术副总裁用一句话概括了整个图景： ![Image](/doc/1228/image/img_b2346cc00837.jpg) 图片来源：NVIDIA 新闻中心 ## 改变一切的 GitHub 仓库现在，事情开始变得真实了。 NVIDIA 不只是发布了模型和文档。他们还发布了一本“食谱”（cookbook）。一个名为 Cosmos Cookbook 的 GitHub 仓库，其中包含分步工作流程、技术配方以及用于构建、适配和部署世界基础模型的完整可运行示例。这不是一篇研究论文。这是可运行的代码，像菜谱书一样组织，专为那些想要今天就动手构建而不是等两年的开发者打造。仓库地址：https://github.com/nvidia-cosmos/cosmos-cookbook 文档网站：https://nvidia-cosmos.github.io/cosmos-cookbook 你可以在 30 秒内克隆它： ``` git clone https://github.com/nvidia-cosmos/cosmos-cookbook.git cd cosmos-cookbook git lfs install # 必需——仓库包含视频文件 ``` ## 里面到底有什么让我带你看看这份 Cookbook 目前包含的内容，因为这些配方相当疯狂。配方 1：自我中心的社会与物理推理该配方使用 Cosmos Reason 2 从第一人称机器人摄像头的视角进行推理。你向它输入一段人类在机器人面前做某事的视频，然后让它解读场景并预测接下来会发生什么。在 NVIDIA 自己的 Cookbook 直播演示中，研究人员展示了一个人的拳头伸向机器人摄像头，并提问：“这个人是在对我做碰拳吗？你能预测出她的拳头在未来 2 秒内最可能处于什么位置吗？” Cosmos Reason 2 回答：是的，这个人正在碰拳。她的拳头可能会保持碰拳姿势，并保持与观察者拳头的接触，直到她收回手势。一个竞争模型则认为这是一个指向手势。一个命令式、强硬的动作。而不是一次互动。一个理解意图的机器人与一个只会看到运动的机器人之间的差距，就是人机交互领域的全部关键所在。Cosmos Reason 2 答对了。而且它只有 80 亿参数。它可以在单张 GPU 上运行。配方 2：机器人导航任务的 Sim-to-Real 迁移 Cookbook 包含一个完整教程，讲解如何对机械臂在一个无菌、虚拟厨房环境中工作的合成机器人仿真视频进行处理。借助 Cosmos Transfer 2.5，它可以生成具有不同橱柜样式、表面材质、光照条件和厨房布局的逼真变体。机器人的几何结构和运动路径被精确保留。但视觉域发生了彻底改变。你最终得到的训练数据看起来像真实的，却无需任何真实世界的数据采集。机器人工程师现在可以从单次仿真中生成多样化的训练数据。一次仿真运行变成十个视觉域。一个环境变成五十个。过去需要现场操作的数据飞轮，现在可以在云端一夜之间运转。配方 3：物理合理性预测这一个更低调，但同样重要。合成数据生成是一种作弊码，直到你不小心让机器人在物理上不可能发生的场景上进行训练。如果你的合成视频显示一个杯子穿过桌子滑行而不是从桌子上弹开，你的机器人就会学到错误的物理规律。这个配方是一个质量过滤器。Cosmos Reason 2 会在生成的视频帧进入训练流水线之前，评估其物理合理性。它会标记物理规律错误的视频片段并自动将其移除。它是为现实本身准备的 QA 层。而且它在你的训练循环看到数据之前就已经运行了。 ## 人们已经在构建什么真实的部署已经在运行了。农业：一家名为 Aigen 的公司正在使用 Cosmos 和 Isaac Sim，在数百万个模拟农场场景中训练自主巡检车以区分作物和杂草。每块田地都不同，因为作物不同、土壤不同、生长阶段不同。Cosmos 在仿真中生成了覆盖所有这些场景的多样性。每台巡检车随后部署在 NVIDIA Jetson Orin 边缘 AI 模块上，在田间进行实时杂草检测决策。原本需要 10,000 小时田间数据采集的工作，现在在一个仿真流水线中完成。 ![Image](/doc/1228/image/img_5dde393efdd7.jpg) 图片来源：aigen.io > **TechCrunch@TechCrunch**：[原文链接](https://x.com/TechCrunch/status/1483861501765165057) > > Aigen 的农业科技机器人集群希望让农业实现碳负排放 > https://techcrunch.com/2022/01/18/aigen-fund-round/… 作者：@Haje 手术机器人：较新的配方之一将 Cosmos Predict 2.5 微调为一个动作条件化的手术模拟器。其成果 Cosmos-H-Surgical-Simulator 是 NVIDIA、约翰·霍普金斯大学以及 35 家机构的合作项目。仅效率提升一项就足以说明问题：40 分钟内完成 600 次策略推演，而真实世界的台面测试需要 2 天。GR00T-H 的原型已经在 SutureBot 基准测试上完成了一次完整的端到端缝合。来源：Hugging Face · Open-H-Embodiment · GitHub > **NVIDIA Healthcare@NVIDIAHealth**：[原文链接](https://x.com/NVIDIAHealth/status/2034038372495167495) > > 想象一下，一个能够像外科医生一样熟练地缝合组织的机器人——将 AI 从感知转变为精确的、挽救生命的行动。 > 探索推动这一飞跃的三大突破： > Open-H-Embodiment —— 首个开放的、社区驱动的、提供标准化的数据集 > > ![图片](/doc/1228/image/img_68145dc4e026.jpg) 工业安全：Cookoff 竞赛吸引了超过 1,600 名参与者。冠军 Zenith 构建了一个用于混合码垛的可解释视觉推理系统，机器人在分拣包裹时会“大声说出”它的推理过程。第二名构建了 ResQ-AI，一个自主灾害响应无人机。来源：NVIDIA 开发者论坛——Cookoff 获奖者 ## 完全的新手今天如何入门你不需要机器人。你不需要仓库。你不需要机器人学博士学位。你只需要一块 GPU 和一个 Hugging Face 账户。如果你没有本地 GPU，你可以在 Nebius 或 Brev 上启动一个云端实例。两者都有可直接启动的 Cosmos 环境。以下是确切的路径。步骤 1：搭建环境（之前已讨论）在克隆之前。如果你使用的是 Mac 或 Windows，请查看 git-lfs.com 上的官方 Git LFS 安装指南对于云端部署，cookbook 文档列出了 Nebius 和 Brev 上开箱即用、预先配置好的 GPU 实例。你完全可以跳过本地环境搭建。第 2 步：运行你的第一次 Cosmos Reason 2 推理这是感受这套技术栈实际能力的最快方式。 Cosmos Reason 2 模型有两种规格：2B 参数和 8B 参数。2B 模型可以在消费级 GPU 上运行。两者都在 Hugging Face 上提供。 ``` from transformers import AutoProcessor, AutoModelForVision2Seq import torch # Load the 2B model — runs on a single consumer GPU model_id = "nvidia/Cosmos-Reason2-2B" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForVision2Seq.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto" ) # Ask it a physics question about a video prompt = """Watch this video carefully. Think step by step about what is physically happening. <think> your reasoning </think> <answer> your answer </answer> Question: What action is being performed and what will happen next?""" ``` 思维链输出（`<think>` 代码块）才是真正的魔力所在。看着它对物理现象、空间关系和可能的结果进行推理。那不是一个分类器。那是一个在实时构建世界模型的模型。第 3 步：尝试自我中心推理（Egocentric Reasoning）配方这是最令人脑洞大开、上手最快的切入点。用第一人称视角录一段短视频（用手机摄像头就行），内容是一个人用手做某件事。向你递一个物体。挥手。捡起某个东西。把它喂给 Cosmos Reason 2，然后问："这个人在做什么？他们接下来要做什么？我应该如何回应？" 模型会通过思维链得出答案，调用空间理解、时间推理和物理直觉。这不仅会给你一个描述，还会给出预测和建议的行动。这就是机器人的大脑。而你刚刚运行了它。完整配方文档在这里： https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html 第 4 步：生成你的第一份合成训练数据如果你能使用 Isaac Sim，甚至只是一个简单的 3D 仿真环境，你就可以使用 Cosmos Transfer 2.5 来生成域适应（domain-adapted）训练数据。工作流程： 1. 对任意机器人任务进行仿真。哪怕是机械臂抓取物体这样的简单场景也可以。 2. 从仿真中导出视频和深度图 3. 启用边缘控制（edge control），输入到 Cosmos Transfer 2.5 4. 指定一个目标视觉域——"工业仓库、荧光灯、混凝土地面"或"家庭厨房、晨光、花岗岩台面" Cosmos Transfer 完全保留机器人的几何结构和运动，但会彻底改变视觉环境。你现在就拥有了看起来像是在真实部署环境中采集的训练数据。 cookbook 中的 sim-to-real 画廊直观展示了输出质量：https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html 第 5 步：深入探索，构建端到端配方当你准备好从单个配方过渡到完整流水线时，cookbook 中包含一个名为 GR00T Dreams 的端到端配方，它将 Cosmos Predict 2.5 和 Cosmos Reason 2 串联在一起。搭建指南：https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html 这个配方能让你从"我理解这套技术栈是做什么的"进阶到"我正在构建一个物理 AI 流水线"。它需要 Ampere 架构的 NVIDIA GPU（RTX Pro 6000、A100、H100）或更新型号，以及 CUDA 12.8+。如果你本地没有这样的条件，就使用云实例。 ## 你应该构建什么这里有五个具体的项目创意，即使没有任何机器人学背景，你本周就可以开始着手。 1. 工作场所安全监控器：使用 VSS 蓝图分析任意环境的摄像头画面——你的办公室、建筑工地、厨房——让 Cosmos Reason 2 描述正在发生的事情、标记潜在危险，并生成每日安全报告。零标注数据。零定制训练。只需提示词。 2. Sim-to-real 数据增强流水线：取任何你可以仿真的 3D 环境（哪怕是一个简单的 Blender 场景），通过 Cosmos Transfer 运行以生成逼真的变体。构建一个按需产出视觉多样性的数据集生成器。 3. 物理合理性过滤器：构建一个数据筛选工具，接收来自任何来源（不仅仅是 Cosmos）的生成视频，并通过 Cosmos Reason 2 检查其物理是否合理。将它作为质量关卡接入任何机器人训练工作流。 4. 自我中心推理 API：将 Cosmos Reason 2 封装在一个简单的 FastAPI 端点中。接受视频上传。返回关于正在发生什么以及接下来应该发生什么的思维链推理。作为推理端点出售给不想自行托管模型的机器人团队。 5. 农业监控概念验证：将摄像头对准任何有植物和物体的户外环境。让 Cosmos Reason 2 随时间识别、分类和跟踪物体。Aigen 的故事表明这可以扩展到工业化农业。它从一台摄像头和一条提示词开始。 ## 为什么现在正是时机让我直说时机的关键。 Cosmos Cookbook 于 2025 年底公开发布。Cookoff 黑客松于今年 1 月至 3 月举行。Cosmos Reason 2 于 2025 年 12 月发布。随后是 GR00T N1.5。Newton 1.0 上周刚刚正式发布（GA）。这套技术栈以目前的形式问世才几周。社区规模还很小。与 18 个月后将会使用它的人数相比，真正克隆过这个仓库并运行过配方的人只是一个零头的零头。在软件领域，ChatGPT 时刻在 2023 年初引发了 LLM 开发者浪潮。那些在 2023 年 1 月就基于 GPT-3.5 进行构建的人领先市场一年。大多数开发者都错过了这个时机。这是同样的转折点。只不过这一次，整个技术栈都是开放的，包括模型、物理引擎、仿真框架、cookbook。NVIDIA 没有设置访问门槛，他们在积极地推动民主化。在这个时刻，存在一种可能性：你是那个早早看清这一切的人。而要成为那样的人，就需要你今天就去克隆这个仓库。 ## 所有资源，一网打尽 Cookbook 主 GitHub 仓库：https://github.com/nvidia-cosmos/cosmos-cookbook Cosmos GitHub 组织完整列表（所有仓库）：https://github.com/nvidia-cosmos Cosmos Reason 2 GitHub：https://github.com/nvidia-cosmos/cosmos-reason2 Cosmos Cookbook 文档站：https://nvidia-cosmos.github.io/cosmos-cookbook Hugging Face 上的 Cosmos Reason 2（2B 模型）：https://huggingface.co/nvidia/Cosmos-Reason2-2B Hugging Face 上的 Cosmos Reason 2（8B 模型）：https://huggingface.co/nvidia/Cosmos-Reason1-7B（搜索 Cosmos Reason 2）无需任何配置即可试用 Cosmos Reason 2：https://build.nvidia.com 自我中心推理（Egocentric Reasoning）配方：https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html Sim-to-Real 机器人画廊：https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html GR00T Dreams 端到端配方：https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html Newton 物理引擎：https://developer.nvidia.com/newton-physics NVIDIA Cosmos 平台概览：https://www.nvidia.com/en-us/ai/cosmos/ 机器人领域的 ChatGPT 时刻，正在此刻发生。整个技术栈都是开放的。去构建点什么吧。 ## 相关链接 - [Utkarsh](https://x.com/utk7arsh) - [@utk7arsh](https://x.com/utk7arsh) - [63K](https://x.com/utk7arsh/status/2048812835514298739/analytics) - [https://github.com/nvidia-cosmos/cosmos-cookbook](https://github.com/nvidia-cosmos/cosmos-cookbook) - [https://nvidia-cosmos.github.io/cosmos-cookbook](https://nvidia-cosmos.github.io/cosmos-cookbook) - [配方 1](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html) - [配方 2](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/transfer1/inference-x-mobility/inference.html) - [配方 3](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/post_training/reason2/physical-plausibility-check/post_training.html) - [aigen.io](https://aigen.io/) - [2022 年 1 月 20 日](https://x.com/TechCrunch/status/1483861501765165057) - [@Haje](https://x.com/Haje) - [Hugging Face](https://huggingface.co/nvidia/Cosmos-H-Surgical-Simulator) - [Open-H-Embodiment](https://huggingface.co/blog/nvidia/physical-ai-for-healthcare-robotics) - [GitHub](https://github.com/NVIDIA-Medtech/Cosmos-H-Surgical) - [3 月 18 日](https://x.com/NVIDIAHealth/status/2034038372495167495) - [7.4K](https://x.com/NVIDIAHealth/status/2034038372495167495/analytics) - [NVIDIA 开发者论坛 —— Cookoff 获奖者](https://forums.developer.nvidia.com/t/the-results-are-in-meet-the-nvidia-cosmos-cookoff-winners-see-them-live-on-april-16/366130) - [git-lfs.com](https://git-lfs.com/) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html) - [https://github.com/nvidia-cosmos/cosmos-cookbook](https://github.com/nvidia-cosmos/cosmos-cookbook) - [https://github.com/nvidia-cosmos](https://github.com/nvidia-cosmos) - [https://github.com/nvidia-cosmos/cosmos-reason2](https://github.com/nvidia-cosmos/cosmos-reason2) - [https://nvidia-cosmos.github.io/cosmos-cookbook](https://nvidia-cosmos.github.io/cosmos-cookbook) - [https://huggingface.co/nvidia/Cosmos-Reason2-2B](https://huggingface.co/nvidia/Cosmos-Reason2-2B) - [https://huggingface.co/nvidia/Cosmos-Reason1-7B](https://huggingface.co/nvidia/Cosmos-Reason1-7B) - [https://build.nvidia.com](https://build.nvidia.com/) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/inference/reason2/intbot_showcase/inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html](https://nvidia-cosmos.github.io/cosmos-cookbook/gallery/robotics_inference.html) - [https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html](https://nvidia-cosmos.github.io/cosmos-cookbook/recipes/end2end/gr00t-dreams/setup.html) - [https://developer.nvidia.com/newton-physics](https://developer.nvidia.com/newton-physics) - [https://www.nvidia.com/en-us/ai/cosmos/](https://www.nvidia.com/en-us/ai/cosmos/) - [升级到 Premium](https://x.com/i/premium_sign_up) - [凌晨 1:13 · 2026 年 4 月 28 日](https://x.com/utk7arsh/status/2048812835514298739) - [63.4K 次浏览](https://x.com/utk7arsh/status/2048812835514298739/analytics) - [查看引用](https://x.com/utk7arsh/status/2048812835514298739/quotes) --- *导出时间：2026/4/28 17:41:24*