Open Source LLMOps Tools
LLMOps covers the practices and software used to build, deploy, evaluate, and maintain applications powered by large language models. It helps teams manage model and prompt changes, connect applications to data and services, assess output quality, and monitor reliability, cost, and security in production. These workflows address the uncertainty and operational demands that can arise when model behavior varies across inputs or updates.
Open source tools in this area include gateways, evaluation frameworks, observability platforms, deployment servers, and security controls. When choosing one, consider its maturity, license, maintenance activity, infrastructure requirements, and compatibility with your models and application stack. LLMOps is useful for developers, researchers, and organizations that need to test and operate language model applications with greater control and transparency.
6 repositories · updated September 4, 2026
Otari: Self-Hosted OpenAI-Compatible LLM Gateway for 40+ Providers
Otari, from Mozilla AI, is an open-source, self-hosted LLM gateway. It provides a single OpenAI-compatible endpoint to connect with over 40 model providers, offering features like virtual keys, budget enforcement, and usage tracking. This solution empowers users to manage their AI stack with greater control and flexibility.
Ragas: Supercharge Your LLM Application Evaluations
Ragas is an ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. It offers objective metrics, intelligent test generation, and data-driven insights to move beyond subjective assessments. This framework helps developers build feedback loops and continuously improve their LLM applications.
LLM Guard: The Security Toolkit for LLM Interactions
LLM Guard is an open-source security toolkit developed by Protect AI, designed to fortify the safety of Large Language Models. It offers comprehensive protection against various threats, including prompt injection, data leakage, and harmful language, ensuring secure and reliable LLM interactions.
langcorn: Serve LangChain LLM Apps and Agents with FastAPI
Langcorn is an innovative API server designed to effortlessly deploy LangChain models and pipelines. It leverages the high-performance FastAPI framework, offering a robust and scalable solution for serving large language model applications. With features like easy installation, built-in authentication, and support for custom API keys, Langcorn streamlines the process of bringing your LLM projects to production.
MCPJungle: Self-Hosted MCP Gateway for AI Agents and Tool Management
MCPJungle is an open-source, self-hosted Model Context Protocol (MCP) Gateway designed for managing AI agents and their tool-calling capabilities. It allows developers and organizations to centralize the registration, discovery, and consumption of MCP servers and their tools, enhancing security and control over AI agent interactions.
Opik: Open-Source LLM Observability, Evaluation, and Optimization
Opik is an open-source platform by Comet designed to streamline the lifecycle of LLM applications. It provides comprehensive tools for debugging, evaluating, and monitoring RAG systems and agentic workflows. Developers can leverage its tracing, automated evaluations, and production-ready dashboards to build and optimize generative AI applications.