Repository History
3 repositories tagged with inference
llm-d Router: Intelligent Orchestration for Multi-Phase LLM Inference
The `llm-d-router` is a sophisticated Go service designed as an intelligent entry point for large language model (LLM) inference requests. It orchestrates complex, multi-phase LLM inference pipelines across specialized worker pools, routing requests through an Inference Gateway to disaggregated vLLM workers. By exposing OpenAI-compatible APIs, it simplifies integration and leverages Kubernetes for scalable, efficient deployments.
Text Generation Inference: High-Performance LLM Serving by Hugging Face
Text Generation Inference (TGI) is a robust toolkit from Hugging Face designed for deploying and serving Large Language Models (LLMs) with high performance. It powers Hugging Face's production services, including Hugging Chat and their Inference API. TGI offers optimized text generation, supporting popular open-source LLMs and implementing advanced features for efficient and scalable inference.
whisper.cpp: High-Performance Speech Recognition with OpenAI's Whisper Model
whisper.cpp is a high-performance C/C++ port of OpenAI's Whisper automatic speech recognition (ASR) model. It offers efficient, dependency-free inference across a wide range of platforms, from desktop to mobile and embedded devices. This project enables fast, local speech-to-text capabilities, making advanced AI accessible for various applications.