microsoft

JARVIS: Connect Language Models to AI Tools

JARVIS, also known as HuggingGPT, uses an LLM to plan requests, choose expert models from Hugging Face, run tasks, and combine their results. It suits research and prototyping that explores natural-language orchestration of AI models.

View on GitHub Compare
Stars
25.4k
Forks
2.2k
Stars, last 30 days
+242

Our take

Health score

49/100

How it is scored
Activity
19
Community
60
Issues
95
Pull requests
20

+242 stars in the last 30 days

  • Closes issues faster than 81% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

25,698

Forks

2,294

Change since Nov 17, 2026: +21 stars·+1 forks

Overview

JARVIS, presented in the HuggingGPT paper, is a research system that connects a language model controller with specialist AI models. It addresses the challenge of routing a natural-language request across different models and combining their outputs into a response.

The workflow plans tasks, selects models based on their descriptions, executes them, and asks the controller to synthesize the results. It is aimed at developers and researchers exploring LLM-based tool use, rather than users seeking a lightweight standalone chatbot.

Key Features

  • Breaks user requests into tasks that can be handled by AI models.
  • Selects expert models hosted on Hugging Face based on model descriptions.
  • Executes selected models and returns their results to the language model controller.
  • Combines task outputs into a final natural-language response.
  • Offers server, web, Gradio, and CLI entry points.
  • Supports local, Hugging Face Inference Endpoint, or hybrid model execution modes.
  • Includes related TaskBench and EasyTool research projects, linked from the README.

Use Cases

  • AI researchers can explore how an LLM can coordinate specialist models across a multi-step request.
  • Developers can prototype workflows that combine tasks such as image understanding, detection, and generation.
  • Teams evaluating task automation can use the related TaskBench project to assess LLM capabilities.
  • Users with suitable infrastructure can run a local or hybrid demo; a hosted Hugging Face Space is also linked from the README.

Getting Started

The README provides server, web, Gradio, and CLI setup instructions. For a lightweight CLI setup, from the server directory run:

python awesome_chat.py --config configs/config.lite.yaml --mode cli

See the repository README for credentials, configuration, model execution modes, and full setup details. The project also links to a Hugging Face Space.

Considerations

  • The controller requires an OpenAI API key, and Hugging Face access may require a token. The README documents both.
  • Local and hybrid execution can require substantial compute and storage. The default configuration lists at least 24 GB of VRAM and more than 284 GB of disk; the lite configuration avoids downloading local expert models but relies on models available through Hugging Face Inference Endpoints.
  • The README describes the code as being in a planning and rebuilding process in a 2023 update. Treat it as a research project and verify current compatibility and endpoint availability before building on it.

Found this useful?

Share it with someone who would like JARVIS.