Repository History
107 repositories tagged with LLM
LLM Compressor: Optimize LLMs for Deployment with vLLM
LLM Compressor is a Transformers-compatible Python library designed to apply various compression algorithms to Large Language Models (LLMs). It enables optimized deployment, especially with vLLM, by offering a comprehensive set of quantization techniques for weights, activations, and KV Cache. This tool seamlessly integrates with Hugging Face models, making LLM optimization accessible and efficient.
LightLLM: A Lightweight and High-Speed LLM Inference and Serving Framework
LightLLM is a Python-based framework designed for efficient Large Language Model (LLM) inference and serving. It stands out for its lightweight architecture, impressive scalability, and high-speed performance, making it an excellent choice for deploying LLMs. The framework integrates and builds upon the strengths of various leading open-source implementations to deliver optimized results.
TensorRT-LLM: Optimizing Large Language Model Inference on NVIDIA GPUs
TensorRT-LLM is an open-source library by NVIDIA designed to optimize inference for Large Language Models (LLMs) and Visual Generation models. It offers a user-friendly Python API, state-of-the-art optimizations, and specialized kernels to ensure efficient performance on NVIDIA GPUs. This powerful tool enables developers to deploy LLMs with high throughput and low latency, from single-GPU setups to multi-node deployments.
DataDreamer: Streamlining Synthetic Data Generation and LLM Workflows
DataDreamer is an open-source Python library designed for efficient prompting, synthetic data generation, and model training workflows. It simplifies the process of creating complex LLM workflows, generating high-quality synthetic datasets, and aligning or fine-tuning models. Built to be simple, efficient, and research-grade, DataDreamer empowers users to build reproducible and shareable AI solutions.
OpenWebAgent: An Open Toolkit for LLM- and LMM-based Web Agents
OpenWebAgent is an open toolkit designed to empower model-based web agents, streamlining human-computer interactions by automating tasks on webpages. It offers a convenient framework for developing LLM- and LMM-based web agents, providing both plugin and server source code for easy integration and customization. This project was featured as an ACL'24 Demo, showcasing its innovative approach to web automation.
LazyLLM: Low-Code Development for Multi-Agent LLM Applications
LazyLLM offers a low-code development tool designed for building multi-agent LLM applications with ease. It simplifies the creation of complex AI applications, providing a streamlined workflow for rapid prototyping, data feedback, and iterative optimization. Developers can leverage its extensive features for deployment, cross-platform compatibility, and efficient model fine-tuning.
Lighteval: Your All-in-One Toolkit for LLM Evaluation
Lighteval is a comprehensive toolkit from Hugging Face for evaluating Large Language Models (LLMs) across various backends. It enables users to dive deep into model performance by saving detailed, sample-by-sample results and supports over 1000 evaluation tasks. The framework offers extensive customization options, allowing users to create custom tasks and metrics tailored to their specific needs.
AgentEvals: Robust Evaluation Tools for LLM Agent Trajectories
AgentEvals is a powerful open-source package from LangChain designed to simplify the evaluation of agentic applications. It provides a collection of ready-made evaluators and utilities, with a particular focus on analyzing agent trajectories, the intermediate steps an agent takes to solve problems. This helps developers understand and improve the reliability and performance of their LLM agents.
Observers: A Lightweight Library for AI Observability in Python
Observers is a Python library designed for AI observability, enabling developers to track and store interactions with generative AI APIs. It provides a flexible framework with various observers for popular LLM providers and multiple storage backends. This tool helps in monitoring, debugging, and analyzing AI model behavior effectively.
XGrammar: Fast, Flexible, and Portable Structured Generation for LLMs
XGrammar is an open-source library for efficient, flexible, and portable structured generation, developed by mlc-ai. It leverages constrained decoding to guarantee 100% structural correctness for outputs like JSON and regex. Optimized for near-zero overhead, XGrammar offers universal deployment across various platforms, hardware, and programming languages, making it a leading solution for structured output from large language models.
FreeLLMAPI: Stack 16 Free LLM Tiers for 1.7 Billion Tokens/Month
FreeLLMAPI is an OpenAI-compatible proxy that aggregates the free tiers of 16 LLM providers, offering access to approximately 1.7 billion tokens per month. It simplifies access to diverse models through a single endpoint, featuring smart routing, automatic failover, and encrypted key storage. This powerful tool is designed for personal experimentation, allowing developers to leverage multiple free LLM resources efficiently.
Hiring Agent: An AI Agent for Resume Evaluation and Scoring
Hiring Agent is an open-source AI agent designed to evaluate and score resumes objectively. It extracts structured data from PDF resumes, enriches it with GitHub profile signals, and provides a fair, explainable evaluation with detailed scores and evidence. This tool supports both local LLMs via Ollama and cloud-based options like Google Gemini.