lance: Store and Query Multimodal Lakehouse Data

Overview

Lance is a file and table format, with a catalog specification, for storing and working with AI and analytics datasets on object storage. It targets a gap in conventional analytics formats: workloads that need efficient random access, multimodal records, and search capabilities alongside columnar scans.

The project includes a Rust core and SDK bindings, including Python. It may suit teams building data pipelines, feature stores, or search systems that want these capabilities within a lakehouse dataset rather than assembling separate storage and indexing systems.

Key Features

• Stores tabular data alongside multimodal content such as images, video, audio, text, and embeddings.
• Supports vector similarity search, BM25 full-text search, and SQL analytics on datasets.
• Provides random-access reads intended for sampling, exploration, and training workloads.
• Supports adding columns with backfilled values without rewriting the full table.
• Provides dataset versioning with ACID transactions, time travel, tags, and branches.
• Integrates with tools including Arrow, Pandas, Polars, DuckDB, Spark, and Ray.
• Offers Rust and Python implementations, with Java bindings also listed in the repository.

Use Cases

• ML platform teams can use Lance datasets as feature stores that combine vector retrieval with structured filters and analytics.
• Data engineers preparing large training datasets can use random access for sampling and can store embeddings or media alongside tabular fields.
• Search developers can build retrieval workflows over vector and text indexes without separating those indexes from the dataset.
• Analytics teams working with images, audio, or video can keep metadata and multimodal data in a shared dataset for exploration and querying.

Project Facts

• Language: Rust
• License: Apache-2.0
• Stars: 7.1k
• Forks: 876
• Topics: apache-arrow, computer-vision, data-analysis, data-analytics, data-centric, data-format, data-science, dataops, deep-learning, duckdb, embeddings, llms, machine-learning, mlops, python, rust
• Archived: No

Getting Started

Install the Python package:

pip install pylance

See the README for conversion, reading, and vector search examples, and the documentation for format and SDK details.

Considerations

• The project is in active development. SDK and API compatibility is separate from file-format compatibility, so check the migration guidance when upgrading.
• For production datasets, select a stable data_storage_version. The README says the next alias is unstable and intended only for experimentation.
• Older Lance releases may not read storage versions introduced by newer releases. Mixed-version deployments should pin the storage version.
• Integrations and SDKs span several ecosystems, so confirm that the required language binding and integrations meet your deployment needs.