files-to-prompt: Concatenate Files into a Single Prompt for LLMs

Summary

files-to-prompt is a command-line tool designed to concatenate the contents of multiple files from a directory into a single, structured prompt suitable for Large Language Models (LLMs). It offers flexible options for filtering files by extension, ignoring patterns, and supports various output formats including standard text, Claude XML, and Markdown. This utility streamlines the process of preparing complex codebases or documentation for AI analysis.

Use at your own risk

OSRepos shares public repositories for knowledge and discovery only. Any installation, execution, configuration, or use of code from these repositories is the user's own responsibility. Always review the repository, source code, dependencies, licenses, and security implications before running or installing anything. OSRepos is not responsible for issues, damages, or losses resulting from third-party repositories.

Introduction

files-to-prompt is a powerful command-line utility created by Simon Willison that simplifies the process of preparing textual data for Large Language Models (LLMs). It efficiently concatenates the contents of multiple files, or even entire directories, into a single, cohesive output. This tool is particularly useful for developers and researchers who need to feed large amounts of code, documentation, or other text files into LLMs for tasks like code analysis, summarization, or question-answering, ensuring the LLM receives a well-structured context.

Installation

Installing files-to-prompt is straightforward using pip:

pip install files-to-prompt

Examples

Here are some common ways to use files-to-prompt:

Basic Usage: Concatenate all files in a directory.

files-to-prompt path/to/directory

Filter by Extension: Include only files with specific extensions.

files-to-prompt path/to/directory -e py -e js

Output as Claude XML: Format the output specifically for Anthropic's Claude LLMs, optimizing for its context window.

files-to-prompt path/to/directory --cxml

Output as Markdown: Generate output with fenced code blocks, useful for pasting into Markdown documents.

files-to-prompt path/to/directory --markdown

Include Line Numbers: Add line numbers to each file's content in the output.

files-to-prompt path/to/directory -n

Ignore Files: Exclude files matching a specific pattern.

files-to-prompt path/to/directory --ignore "*.log"

Read from Stdin: Pipe file paths from another command, such as find.

find . -name "*.py" -print0 | files-to-prompt --null

Why Use files-to-prompt?

In the era of powerful LLMs, providing well-structured and relevant context is crucial for obtaining accurate and useful responses. files-to-prompt addresses this need by:

Streamlining Context Preparation: Automatically gathers and formats content from multiple files, saving manual effort.
Optimizing for LLMs: Offers specific output formats like Claude XML and Markdown, which can improve how LLMs process and understand the provided context.
Flexibility and Control: Provides extensive options for filtering, ignoring, and customizing the output, allowing users to precisely control what content is included.
Developer-Friendly: Integrates seamlessly into existing workflows, especially for projects involving codebases or extensive documentation.