torchchat: Run PyTorch LLMs on Desktop, Server, and Mobile

Overview

torchchat provides a way to run supported large language models with PyTorch across Python environments, desktop and server systems, and mobile devices. It brings model downloads, chat and text generation, export, and evaluation into one toolkit, addressing the challenge of adapting local inference to different execution environments.

It is intended for developers experimenting with or integrating local LLM inference, especially where PyTorch-native execution or deployment through AOT Inductor and ExecuTorch is useful. The project is no longer under active development, so it is best treated as an existing toolkit rather than a growing platform.

Key Features

• Run interactive chat and prompt-based text generation from Python.
• Serve chat completions through a locally hosted REST API with an OpenAI-style interface.
• Launch a basic browser chat interface against a local server.
• Export models for AOT Inductor or ExecuTorch, with Python and C++ runner options.
• Deploy exported models to iOS and Android using ExecuTorch.
• Use supported model aliases spanning Llama, Mistral, Granite, DeepSeek, and other listed families.
• Configure data types and quantization, including mobile-oriented quantization options.
• Evaluate models through an integration with lm-evaluation-harness.

Use Cases

• A PyTorch developer wants to test local chat or text generation without building an inference stack from scratch.
• An application developer wants to export a supported model and run it from a C++ desktop or server process.
• A mobile developer wants to explore on-device LLM inference using ExecuTorch and an exported model artifact.
• A team comparing model outputs or configurations wants to run the repository's evaluation workflow on a supported model.

Project Facts

• Language: Python
• License: BSD-3-Clause
• Stars: 3.6k
• Forks: 248
• Topics: llm, local, pytorch
• Archived: true

Getting Started

The README specifies Python 3.10 and recommends using a virtual environment. A minimal setup is:

git clone https://github.com/pytorch/torchchat.git
cd torchchat
python3 -m venv .venv
source .venv/bin/activate
./install/install_requirements.sh

See the README for model access, usage commands, and platform-specific deployment instructions.

Alternatives

• cactus: Cactus is a C++-focused engine for local language, vision, and speech inference, while torchchat offers PyTorch, C++, and mobile execution paths for language models.

Considerations

• The repository states that it is no longer under active development, and it is archived. Expect limited maintenance and no assumption of ongoing fixes.
• The README recommends Python 3.10 and a virtual environment. Mobile deployment also requires setting up ExecuTorch and the relevant iOS or Android development tools.
• Many model weights are distributed through Hugging Face. Some models require account access or additional authorization, and third-party model terms may apply.
• The REST server and evaluation workflow are identified in the README as works in progress, with some parameters or features not fully implemented.
• Model size, hardware compatibility, and performance vary. The README does not guarantee performance or compatibility.