dlt: Load Data from Sources into Analytics Destinations

Overview

dlt is a Python library for moving data from varied sources into structured datasets in databases, warehouses, and file-based destinations. It addresses the repeated plumbing involved in pagination, schema discovery, type conversion, and adapting to changes in incoming data.

It is designed to run inside existing Python workflows rather than as a separate data platform. Teams can use it for lightweight pipelines in scripts or notebooks, or integrate it with orchestration they already use. It is most useful when you want declarative loading and schema handling without building those mechanisms from scratch.

Key Features

• Extract from REST APIs, SQL databases, filesystems, pandas and Polars DataFrames, Arrow tables, and Python iterables.
• Load pipelines to destinations including DuckDB, BigQuery, Snowflake, PostgreSQL, and cloud filesystems.
• Infer schemas and normalize nested data into typed tables.
• Support incremental loads, merge strategies, primary keys, and schema contracts.
• Manage destination credentials through configuration files or environment variables.
• Attach to existing pipelines and read data back through DataFrame, Arrow, or Ibis interfaces.
• Adapt to schema drift by creating or altering destination tables as needed.

Use Cases

• A data engineer can pull paginated API data into a warehouse without implementing request and pagination handling for each endpoint.
• An analyst or developer can load local files or Python-generated records into DuckDB for notebook exploration.
• A team can incrementally replicate database tables to reduce the need to reload unchanged records.
• A data platform team can standardize ingestion across different sources while retaining its existing Python and orchestration workflows.

Getting Started

Install the library with the destination extra you need:

pip install "dlt[duckdb]"

See the README and official documentation for setup and pipeline examples.

Considerations

• dlt is a library, not a hosted platform. You run and operate pipelines in your own environment.
• Destination support may require optional extras and configuration for credentials and services.
• Python 3.14 support is described as experimental because some optional extras are not yet available.
• The project notes that new destinations are unlikely to be merged because of their maintenance cost.