pdf-inspector: Fast Rust Library for PDF Classification and Text Extraction

Introduction

pdf-inspector is a high-performance Rust library developed by Firecrawl, designed for efficient PDF inspection, classification, and text extraction. It intelligently distinguishes between scanned and text-based PDFs, enabling smart routing decisions that can significantly reduce reliance on expensive OCR services. This library offers robust capabilities for converting PDF content into clean Markdown, complete with structured elements like headings, lists, and tables. Beyond its native Rust interface, pdf-inspector provides convenient bindings for Python, Node.js, and browser WebAssembly, making it accessible across various development environments. Its impressive benchmark results highlight its speed and accuracy in handling complex PDF layouts.

Installation

Getting started with pdf-inspector is straightforward across its supported platforms.

Python

pip install pdf-inspector

Node.js

npm install @firecrawl/pdf-inspector

Browser WebAssembly

npm install @firecrawl/pdf-inspector-wasm

Rust

Add to your Cargo.toml:

[dependencies]
pdf-inspector = "1"

Or install via cargo add:

cargo add pdf-inspector

CLI

cargo install pdf-inspector

For full API references, please consult the official documentation:

• Python API Reference
• Node.js API Reference
• Browser WebAssembly API Reference
• Rust API Reference

Examples

Here are quick examples demonstrating how to use pdf-inspector for common tasks.

Python

import pdf_inspector

result = pdf_inspector.process_pdf("document.pdf")
print(result.pdf_type)   # "text_based", "scanned", "image_based", "mixed"
print(result.markdown)   # Markdown string or None

# Selective OCR; clean text PDFs do not load the external OCR runtime.
ocr = pdf_inspector.process_pdf_with_ocr("document.pdf")
print(ocr.pages_routed_to_ocr)

Node.js

import { readFileSync } from 'fs';
import { processPdf, processPdfWithOcr } from '@firecrawl/pdf-inspector';

const pdf = readFileSync('document.pdf');
const result = processPdf(pdf);
console.log(result.pdfType);   // "TextBased", "Scanned", "ImageBased", "Mixed"
console.log(result.markdown);  // Markdown string or null

const ocr = await processPdfWithOcr(pdf); // selective OCR, off the event loop
console.log(ocr.pagesRoutedToOcr);

Browser WebAssembly

import init, { processPdf } from '@firecrawl/pdf-inspector-wasm';

await init();
const response = await fetch('/document.pdf');
const pdf = new Uint8Array(await response.arrayBuffer());
const result = processPdf(pdf);

console.log(result.pdfType);
console.log(result.markdown);

Rust

use pdf_inspector::process_pdf;

let result = process_pdf("document.pdf")?;
println!("Type: {:?}", result.pdf_type);
if let Some(markdown) = &result.markdown {
    println!("{}", markdown);
}

CLI

# Convert PDF to Markdown
pdf2md document.pdf

# JSON output (for piping)
pdf2md document.pdf --json

# Detection only (no extraction)
detect-pdf document.pdf

Why Use pdf-inspector?

pdf-inspector stands out for its unique combination of speed, accuracy, and versatility, making it an ideal choice for various PDF processing needs.

• Intelligent PDF Classification: It quickly identifies PDF types (TextBased, Scanned, ImageBased, Mixed) with high confidence, allowing for efficient routing. This smart classification saves significant costs and latency by avoiding unnecessary OCR for text-based documents.
• High-Fidelity Text Extraction: The library performs position-aware text extraction, capturing font information, X/Y coordinates, and automatically handling multi-column reading order. This ensures accurate and structured text output.
• Comprehensive Markdown Conversion: pdf-inspector excels at converting complex PDF layouts into clean Markdown. It intelligently detects and formats headings (H1-H4), various list types, code blocks, tables (both rectangle-based and heuristic), bold/italic text, and URLs, providing a highly readable output.
• Multi-Platform Accessibility: With robust bindings for Python, Node.js, and browser WebAssembly, alongside a powerful Rust API and CLI tools, pdf-inspector integrates seamlessly into diverse development workflows.
• Exceptional Performance: Benchmarks demonstrate pdf-inspector's superior performance in overall score, reading order, table detection, and processing speed compared to other local PDF parsers. It processes 200 documents in under half a second, making it incredibly fast.
• Cost and Latency Savings: By accurately identifying text-based PDFs, pdf-inspector allows developers to process a majority of documents locally and quickly, bypassing the need for expensive and time-consuming external OCR services. This is particularly beneficial for reports, research papers, financial documents, and legal PDFs.

Links

• GitHub Repository: https://github.com/firecrawl/pdf-inspector
• Crates.io: https://crates.io/crates/pdf-inspector
• npm Package: https://www.npmjs.com/package/@firecrawl/pdf-inspector
• PyPI Package: https://pypi.org/project/pdf-inspector/
• License (MIT): https://github.com/firecrawl/pdf-inspector/blob/main/LICENSE