Open Source Data Processing Projects
Data processing covers the methods and software used to collect, clean, validate, transform, combine, and organize information. These tasks turn raw or inconsistent inputs into data that people and applications can analyze or use. Tools in this area help automate repeatable workflows, handle large volumes, convert between formats, and catch errors before they affect downstream systems. They can work with structured records, documents, spreadsheets, or streams of incoming data.
Open source options include command-line utilities, libraries, workflow orchestrators, format converters, and validation tools. When choosing one, consider supported formats and scale, compatibility with your existing systems, setup and runtime requirements, license, documentation, and maintenance activity. Some tools are suited to quick, local transformations, while others support scheduled or distributed pipelines. They are useful to developers, analysts, researchers, and teams that need reliable, repeatable ways to prepare and move data.
10 repositories · updated August 30, 2026
csvkit: Powerful Command-Line Tools for CSV Data Manipulation
csvkit is a robust suite of command-line utilities designed for efficient conversion and manipulation of CSV files. It simplifies complex data tasks, making it an indispensable tool for anyone working with tabular data. Inspired by tools like pdftk and GDAL, csvkit brings powerful data processing capabilities directly to your terminal.
Argo Workflows: A Cloud-Native Workflow Engine for Kubernetes
Argo Workflows is an open-source, container-native workflow engine designed for orchestrating parallel jobs on Kubernetes. It allows users to define multi-step workflows where each step is a container, modeling dependencies using directed acyclic graphs (DAGs). This CNCF graduated project is ideal for machine learning pipelines, data processing, and CI/CD.
DataTrove: Streamlining Large-Scale Data Processing for LLMs
DataTrove is a powerful Python library designed to simplify the complex task of processing, filtering, and deduplicating text data at a massive scale. It offers a collection of customizable, platform-agnostic pipeline blocks, making it ideal for preparing training data for large language models. With support for various execution environments, DataTrove frees developers from scripting madness, enabling efficient and reproducible data workflows.
CyToolz: High-Performance Functional Utilities for Python
CyToolz is a high-performance Cython implementation of the popular Toolz library, offering functional utilities for iterables, functions, and dictionaries. It provides significant speed improvements, typically 2-5x faster than its pure Python counterpart, while maintaining the same API. This makes CyToolz an excellent choice for data processing tasks requiring efficiency and low memory usage.
PinescriptV6-docs-crawler: Python Tool for Pine Script V6 Documentation
PinescriptV6-docs-crawler is a Python tool designed to crawl and process TradingView's Pine Script V6 documentation. Utilizing the Crawl4Ai framework, it efficiently extracts, cleans, and organizes this documentation into searchable markdown files. This makes it significantly easier for developers to reference and analyze Pine Script features and syntax.
GraphRAG: A Modular Graph-Based RAG System for LLM Discovery
GraphRAG, developed by Microsoft, is a powerful and modular graph-based Retrieval-Augmented Generation (RAG) system. It is designed to extract meaningful, structured data from unstructured text using Large Language Models (LLMs). This system enhances an LLM's ability to reason about private and narrative data by leveraging knowledge graph memory structures.
Cerberus: Lightweight and Extensible Data Validation for Python
Cerberus is a lightweight and extensible data validation library for Python, offering robust type checking and base functionality. It is designed for easy customization and integration, allowing for custom validation rules. With no external dependencies, Cerberus provides a powerful yet simple solution for validating data structures.
Attachments: The Python Funnel for LLM Context and Multimodal Data
Attachments simplifies providing context to Large Language Models by transforming various file types into model-ready text and images. This Python library acts as a universal funnel, enabling developers to integrate diverse data sources like PDFs, images, web content, and even entire code repositories with just a few lines of code. It supports popular LLM APIs and frameworks, making multimodal AI development more accessible.
Dasel: A Universal CLI Tool for Data Selection and Transformation
Dasel is a powerful command-line tool and Go library designed for querying, modifying, and transforming data across various formats like JSON, YAML, TOML, XML, and CSV. It provides a consistent syntax, making it an invaluable asset for developers, DevOps professionals, and anyone involved in data wrangling tasks. Its ability to convert between formats and integrate into scripts further enhances its utility.
JSON For You: The Ultimate JSON Visualization and Processing Tool
JSON For You is a powerful web-based tool designed for comprehensive JSON visualization and processing. It offers various view modes like Graph and Table, supports structured comparisons, and integrates `jq` for advanced querying. This open-source project provides an intuitive UI for developers working with JSON data.