TurboLLM: Run Local Language Models With GPU Tuning

Overview

TurboLLM is a local LLM runtime and web app that manages model downloads, inference engines, and API access from one place. Its main distinction is that it can use different llama-server-compatible engines, rather than limiting users to a single bundled runtime, and can benchmark and tune settings for the available hardware.

It is aimed at people who want to run models on their own machine and connect them to existing tools or coding agents. The web UI and compatible APIs make it useful beyond interactive chat, while its engine flexibility is especially relevant when a fork supports features unavailable in standard runtimes.

Key Features

• Browser-based interface for downloading, loading, and chatting with local models.
• OpenAI-compatible and Anthropic-compatible APIs, including support for Claude Code integrations.
• Support for custom llama-server-compatible engines and selected engine types such as MLX and vLLM.
• Hardware-aware benchmarking, load-parameter tuning, and measured generation speeds.
• Model library support for existing GGUF folders and Hugging Face downloads.
• Command-line launcher and a local gateway that can load requested models on demand.
• Runs on Windows, macOS, and Linux, with CPU fallback when a GPU is unavailable.

Use Cases

• Developers can connect Claude Code or other compatible tools to models running on their own hardware.
• Local-LLM users can try community engine forks without replacing their full workflow or manually tuning every launch flag.
• Teams can expose a local inference API to tools on a LAN, with API-key authentication enabled when sharing.
• Users with existing GGUF collections can manage and load those models without downloading them again.

Project Facts

• Language: TypeScript
• Stars: 278
• Forks: 40
• Topics: ai, anthropic-api, claude-code, gguf, gpu, inference, llama-cpp, llama-server, llm, local-llm, offline, openai-api, self-hosted
• Archived: No

Getting Started

With Node.js 22.13.0 or newer, run:

npx turbollm

See the repository README for setup, engine options, configuration, and API details.

Alternatives

• LocalAI: LocalAI emphasizes a broad API-serving stack for language, vision, audio, and image models, while TurboLLM focuses on local language models with hardware-aware tuning and engine choice.
• textgen: TextGen centers on a private desktop and web UI for managing and using local models, while TurboLLM emphasizes hardware-aware tuning and OpenAI- or Anthropic-compatible APIs.

Considerations

TurboLLM requires Node.js and a terminal command to start, so it is not currently a one-click desktop installer. A GPU is recommended for accelerated inference, though the README describes a CPU fallback. Some capabilities, including remote access and Turbo Link, are marked experimental. The README identifies the license as FSL-1.1 with a future Apache-2.0 grant; this is source-available rather than an unrestricted permissive license, so review its terms before redistribution or commercial product use.