sglang-omni: Serve Interactive Multimodal Models with SGLang

Overview

SGLang-Omni is a multi-stage serving runtime built around SGLang, the serving and inference framework from the SGLang project. It targets interactive intelligence systems, where generation may span different model stages with distinct compute and communication needs, rather than a single autoregressive model call.

The runtime coordinates those stages and provides OpenAI-compatible endpoints for multimodal chat, speech generation, and transcription. CUDA is the default supported backend; Apple Silicon and Intel GPUs are listed as experimental options. It is aimed at teams serving speech and multimodal models who need streaming or multi-stage execution and want to compose with SGLang for autoregressive scheduling where applicable.

Key Features

• Coordinates preprocessing, encoders, autoregressive engines, decoders, vocoders, and other stages in one runtime.
• Uses stage-specific schedulers for workloads such as autoregressive generation and streaming vocoder loops.
• Moves tensor payloads between stages through shared-memory, NCCL, NIXL, and Mooncake transport backends.
• Exposes OpenAI-compatible endpoints for multimodal chat, speech generation, batch and streaming speech, uploaded voices, and transcription.
• Supports model families for omni chat, text-to-speech, music generation, audio transcription, and diarization.
• Includes a router for multi-worker health, readiness, lifecycle, and capability discovery.
• Supports NVIDIA CUDA by default, with experimental paths for Apple Silicon and Intel XPU.

Use Cases

• Inference platform teams can serve streaming text-to-speech models behind an OpenAI-compatible audio API.
• Teams building voice interfaces can combine audio input, transcription, and multimodal chat in a staged runtime.
• Speech product developers can deploy transcription and diarization models through a shared serving surface.
• Research and infrastructure teams working with music or omni models can coordinate heterogeneous model stages and transport between them.

Getting Started

The README links to the installation guide and model-specific usage guides. For the documented prerelease installation, use:

uv pip install --prerelease=allow "sglang-omni==0.1.9"

See the installation guide and repository README for setup details and model-specific instructions.

Alternatives

• chatterbox-vllm: Chatterbox-vllm serves one TTS model through vLLM, while SGLang-Omni coordinates stages for a broader range of speech and multimodal models.

Considerations

• The project is still young, even though its recent commit and release activity is regular. Treat model and backend support as something to validate against your own workloads.
• The default backend is NVIDIA CUDA. Apple Silicon and Intel XPU are explicitly experimental, so platform-specific behavior may require additional testing.
• The runtime involves multiple stages, schedulers, and transport backends. Operating it may require more systems integration than serving a single model.
• The repository has substantial open issue activity, while recent issue closures are limited; assess whether your deployment needs depend on fixes or support that are not yet available.
• The project is licensed under Apache-2.0. Review the license terms and any obligations relevant to your distribution or deployment.