PinescriptV6-docs-crawler: Crawl and Chunk Pine Script Docs

Overview

PinescriptV6-docs-crawler automates collecting TradingView’s official Pine Script v6 documentation and preparing it for downstream use. It converts documentation pages into cleaned Markdown, then can split them into sections with heading context for retrieval systems.

It is useful for developers building searchable references or RAG collections around Pine Script. The repository produces files for ingestion; it is not itself a vector database or a Pine Script editor.

Key Features

• Crawls the official Pine Script v6 documentation with Crawl4AI.
• Uses SHA-256 hashes in a manifest to skip unchanged pages on later runs.
• Cleans and overwrites Markdown files instead of creating timestamped copies.
• Splits content at H1 and H2 headings into chunks under 3,000 characters, with 200-character overlap.
• Preserves Pine Script code blocks when chunking.
• Adds section breadcrumbs to chunks for retrieval context.
• Writes a combined JSONL file for one-pass ingestion into a vector database.

Use Cases

• Pine Script developers can build a searchable local reference when working with language syntax and features.
• RAG developers can prepare Pine Script documentation chunks for ingestion into a vector store.
• Documentation maintainers can refresh a local copy after changes to the official documentation.
• Teams evaluating Pine Script support in coding assistants can use the generated Markdown and JSONL as source material.

Getting Started

Install the dependencies and browser runtime, then run the full pipeline:

pip install -r requirements.txt
playwright install
python crawl_and_process.py

See the README for individual pipeline steps, testing, and refresh workflow details.

Alternatives

• AnyCrawl: A general-purpose TypeScript crawler for varied sites and LLM extraction, rather than a Pine Script v6 documentation crawler producing heading-aware Markdown chunks.
• trafilatura: A general-purpose Python web content extractor and crawler, rather than a Pine Script v6-specific pipeline with incremental hashing and heading-aware chunks.

Considerations

• Crawling requires the project dependencies and Playwright browser installation.
• The output is intended for downstream search or RAG ingestion. Connecting to and maintaining a vector database is outside the described pipeline.
• Refreshing is manual by default through a GitHub Actions workflow, so documentation updates are not automatically checked on a schedule.