Repository History
8 repositories tagged with nlp
Axolotl: Streamlining LLM Fine-tuning with a Powerful Open-Source Framework
Axolotl is a comprehensive, free, and open-source framework designed to simplify the post-training and fine-tuning processes for large language models (LLMs). It offers extensive model support, diverse training methods, and robust performance optimizations, making it an invaluable tool for researchers and developers. With easy configuration and cloud-ready deployment, Axolotl empowers users to efficiently customize and enhance LLMs.
Docling: Streamline Document Processing for Generative AI Applications
Docling is a powerful Python library designed to simplify document processing, preparing diverse formats for generative AI applications. It offers advanced parsing capabilities, including sophisticated PDF understanding, and provides a unified document representation. With seamless integrations into the AI ecosystem, Docling empowers developers to build robust AI solutions.
LangTest: A Comprehensive Library for Safe & Effective Language Models
LangTest is an open-source Python library dedicated to ensuring the safety and effectiveness of language models. It offers a comprehensive framework for testing model quality, covering robustness, bias, fairness, and accuracy across various NLP tasks and LLM providers. With LangTest, developers can generate and execute over 60 distinct test types with just one line of code, promoting responsible AI development.
Unstructured: Open-Source Pre-Processing for Complex Document Data
The `unstructured` library is an open-source ETL solution designed to convert complex, unstructured documents into clean, structured data. It streamlines the data processing workflow for language models, offering tools for ingesting and pre-processing various document types like PDFs, HTML, and Word documents. This library simplifies the transformation of raw information into formats suitable for advanced AI applications.
DataTrove: Streamlining Large-Scale Data Processing for LLMs
DataTrove is a powerful Python library designed to simplify the complex task of processing, filtering, and deduplicating text data at a massive scale. It offers a collection of customizable, platform-agnostic pipeline blocks, making it ideal for preparing training data for large language models. With support for various execution environments, DataTrove frees developers from scripting madness, enabling efficient and reproducible data workflows.
EasyEdit: An Easy-to-Use Knowledge Editing Framework for LLMs
EasyEdit is an open-source framework designed for efficient knowledge editing in Large Language Models (LLMs). It provides a unified, easy-to-use platform to modify, insert, or erase specific knowledge within LLMs without negatively impacting overall performance. This tool is crucial for aligning LLMs with evolving user needs and correcting factual inaccuracies.
Text Generation Inference: High-Performance LLM Serving by Hugging Face
Text Generation Inference (TGI) is a robust toolkit from Hugging Face designed for deploying and serving Large Language Models (LLMs) with high performance. It powers Hugging Face's production services, including Hugging Chat and their Inference API. TGI offers optimized text generation, supporting popular open-source LLMs and implementing advanced features for efficient and scalable inference.
FinGPT: Open-Source Large Language Models Revolutionizing Financial AI
FinGPT is an open-source initiative providing large language models specifically designed for the financial sector. It aims to democratize access to advanced AI for finance, offering cost-effective and adaptable solutions for tasks like sentiment analysis and robo-advisory, addressing the dynamic nature of financial data.