ctx-gate: LLM Context Gateway for Efficient Token Usage

Introduction

ctx-gate is an innovative LLM-agnostic context optimization proxy that sits between your development environment or tool and any Large Language Model. Its primary goal is to automatically reduce session token consumption without sacrificing the essential facts your prompts rely on, all while requiring no changes to your existing workflow. This powerful gateway supports both Anthropic Messages API and OpenAI-compatible endpoints, making it versatile for various LLM-powered applications like Claude Code, Cursor, Continue.dev, and any OpenAI SDK.

Why Use and Key Benefits

LLM coding tools often consume tokens rapidly due to several factors, including compounding context, verbose tool outputs, full file reinjection, task bleed, and model overkill. ctx-gate addresses these challenges transparently at the proxy layer through its modular architecture:

• Task Shift Detector: Automatically clears context when a new task is detected, carrying forward only key facts to the new session's system prompt.
• Context Compressor: Applies various strategies, such as rolling summaries of old turns, relevance-scored retention of critical facts, file diff injection, and tool output truncation. This module ensures significant token savings while maintaining 100% fact retention, as verified by its built-in faithfulness harness.
• Model Router: Dynamically selects the appropriate model tier (fast, standard, advanced) based on prompt complexity, optimizing both cost and performance.
• Checkpoint Writer: Saves session state for restart recovery, injecting the last checkpoint into new system prompts automatically.

The project also includes a robust faithfulness evaluation harness, which measures token savings and information retention, providing concrete evidence of its effectiveness.

Installation

Getting started with ctx-gate is straightforward. You can install it via pip:

pip install ctx-gate

For development or to track the main branch, you can install from source:

git clone https://github.com/nac7/ctx-gate
cd ctx-gate
pip install -e .

Optional extras for enhanced functionality:

pip install tiktoken                      # for exact token counts
pip install "ctx-gate[rag]"               # for RAG retrieval + embedding relevance

Examples

Once installed, you can start the ctx-gate proxy and point your LLM tools to it.

Start the proxy:

# Claude (default)
ANTHROPIC_API_KEY=sk-ant-... python ctx_gate.py serve --verbose

# OpenAI
OPENAI_API_KEY=sk-... python ctx_gate.py serve --provider=openai

# Local Ollama (no key needed)
python ctx_gate.py serve --provider=ollama

# Custom port
python ctx_gate.py serve --port=9000

Point your tool at ctx-gate:

• Claude Code (~/.claude/settings.json):

  {
        "env": {
          "ANTHROPIC_BASE_URL": "http://127.0.0.1:8080"
        }
      }

• Cursor / Continue.dev / VS Code: Change the API base URL to http://127.0.0.1:8080/v1.
• Any OpenAI SDK:

  from openai import OpenAI
  client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="any")

Links

• GitHub Repository: https://github.com/nac7/ctx-gate
• PyPI: https://pypi.org/project/ctx-gate/
• DOI: https://doi.org/10.5281/zenodo.21877377