Isaac-GR00T: Adapt Vision-Language Models for Robot Control

Overview

NVIDIA Isaac GR00T is a vision-language-action model and reference toolkit for robots. It combines visual and language input with action generation to support manipulation and other skills across different robot embodiments.

The project is aimed at robotics researchers and developers who need to evaluate a pretrained policy, adapt it to their own robot demonstrations, or connect inference to simulation or hardware. Its workflows cover data preparation, fine-tuning, evaluation, and deployment.

Key Features

• Pretrained GR00T N1.7 checkpoints for zero-shot inference and task-specific adaptation.
• Multimodal policy input, including language and images, with continuous action prediction.
• Fine-tuning support for custom robot data and modality configurations.
• Dataset handling based on LeRobot v2, extended with GR00T modality metadata.
• Open-loop evaluation and server-client policy serving for simulation or robot integration.
• Deployment tooling that includes ONNX and TensorRT export support.
• Examples and workflows for humanoid and other robot embodiments, including LIBERO, DROID, SimplerEnv, and SO100.

Use Cases

• Robotics researchers can test whether a pretrained policy transfers to a new task or embodiment before collecting extensive task-specific data.
• Robot developers can fine-tune the model on demonstrations from their own robot using a compatible dataset and modality configuration.
• Benchmark teams can reproduce or evaluate policies on supported simulation tasks such as LIBERO and SimplerEnv.
• Humanoid robotics teams can explore language-conditioned manipulation and whole-body control workflows with supported controllers.

Project Facts

• Language: Python
• License: Apache-2.0
• Stars: 8.2k
• Forks: 1.5k
• Topics: none listed
• Archived: no

Getting Started

Clone the repository with its submodules, then install the environment with uv:

git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
uv sync --python 3.12

The model backbone requires Hugging Face access. See the README for platform-specific setup, inference, and fine-tuning instructions.

Considerations

• Inference requires a GPU with 16 GB or more VRAM; fine-tuning recommends one or more GPUs with 40 GB or more VRAM.
• The backbone is a gated Hugging Face model, so users must request access and authenticate before inference or fine-tuning.
• Setup varies by platform and depends on compatible CUDA, Python, and video-decoding dependencies. The README notes that the default torchcodec setup supports FFmpeg versions 4 through 7, not FFmpeg 8.
• The code is licensed under Apache 2.0, but model weights use the separate NVIDIA Open Model License.
• Adapting the policy requires robot demonstrations in the expected dataset format and a matching embodiment or modality configuration.