lance-format

lance-format/lance

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

6.8K stars 760 forks 6.8K watchers Rust Apache License 2.0

apache-arrow computer-vision data-analysis data-analytics data-centric data-format data-science dataops deep-learning duckdb embeddings llms machine-learning mlops python rust

View on GitHub Website

77 Open Issues Need Help Last updated: Jul 14, 2026

Open Issues Need Help

View All on GitHub

refactor: use LazyLock to initialize Field and Schema once about 22 hours ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support StringView type in FTS indices 1 day ago

good first issue A-index

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add `slice()` method to `LanceDataset` 23 days ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Clean up `copy` `rename` logic about 1 month ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

working with pydantic model instances about 2 months ago

enhancement good first issue A-python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Expose `io_buffer_size` in Java in `CompactionOptions` about 2 months ago

good first issue A-java

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add `n_jobs` parameter to `lance.write_dataset` to speed up writing large in-memory tables about 2 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Typecheck `lance/arrow.py` and `tests/test_arrow.py` about 2 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make a SVD implementation in Rust and remove the dependency of openblas about 2 months ago

enhancement good first issue help wanted rust A-index

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow filtering of projected fields about 2 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Zero dimension vectors cause Lance to panic about 2 months ago

bug good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

CompactionMetrics doesn't have a repr in Python about 2 months ago

good first issue A-python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

GPU-enabled index training about 2 months ago

good first issue A-python PyTorch

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Display warning if the index is not built for a vector column during query about 2 months ago

good first issue help wanted A-python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

implement existing V2 indices as `VectorIndexExtension` about 2 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow issue vector search in batch mode about 2 months ago

enhancement good first issue help wanted A-python rust A-index

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow dictionary encoding to support more fixed width types 3 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Example converter for nuscenes dataset 3 months ago

good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Support predicate pushdown via pyarrow dataset 3 months ago

good first issue help wanted duckdb benchmark rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support Storage Options in Spark Connector 3 months ago

good first issue feature

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Don't materialize any data on `count_rows` if the filter can be satisfied by a scalar index 4 months ago

good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feature: support datafusion expression for row deletions 4 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Parametrize SQL strings 5 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Can't filter for vector indices in `describe_indices` 5 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add `ReadFileError` and `DecodeColumnError` wrappers to provide better context for decoding errors 5 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Write specification doc describing schema format 6 months ago

documentation good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Column Statistic improvment list after MVP 6 months ago

enhancement help wanted feature file-storage format

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add general compression for large block arrays 7 months ago

AI Summary: The issue proposes adding general compression capabilities for large block arrays, specifically targeting large dictionary arrays. Currently, block compression is limited to bitpacking, and this enhancement aims to improve support and efficiency for data types that are not well-suited for bitpacking.

Complexity: 3/5

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Add optimizer rule to change `LanceScanExec -> FilterExec` into `FilteredReadExec` 7 months ago

good first issue performance

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Remove duplicate datafusion code 7 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Enhance not found errors with suggestions 7 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Review use of `Error::io()` to find cases where that error should be a different type 7 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Offset overflow errors can be confusing for users 8 months ago

good first issue file-storage

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

opening non-existent dataset should return `Error::NotFound` instead of `Error::IO` 8 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

chore: consolidate Dataframe ProjectExec and lance ProjectExec 8 months ago

good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Tracking Lance Datasets (DNC) 8 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

enable code coverage for python tests 8 months ago

enhancement good first issue help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

`LanceFragment.to_batches` not respecting `filter` kwarg 8 months ago

bug good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow relative paths in `IvfModel.save()` and `PqModel.save()` 8 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

All binary array tests generate fixed size binary arrays 8 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Exclude sign bit from bitpacked encoding if all values are negative 8 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Fragment::create_from_file not supported with v2 files 8 months ago

help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make it possible to pass `storage_options` to `LanceFileReader` and `LanceFileWriter` 8 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

lance.torch.data.LanceDataset(to_tensor_fn=...) typehint inconsistent with usage. 8 months ago

good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

pylance write to local disk with a path started with `s3://` 8 months ago

bug good first issue help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Support export table from DuckDB to lance 8 months ago

enhancement help wanted duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Implement replacement scan function for Lance Scan in DuckDB 8 months ago

good first issue help wanted duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Create pylance build for conda environment 8 months ago

good first issue help wanted python ci

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Expose vector search capability via duckdb extension 8 months ago

good first issue help wanted duckdb rust vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DuckDB] Filters pushdown via native lance reader (`lance_scan`) 8 months ago

enhancement good first issue duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support version metadata when writing a dataset 8 months ago

enhancement help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[DOC] Update readme to list the cloud storage supported 8 months ago

documentation good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

[Duckdb] Zero copy Arrow Array/RecordBatch to duckdb Vector/DataChunk conversion 8 months ago

help wanted duckdb arrow rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support Arrow Tensor type 8 months ago

enhancement help wanted python arrow rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Python: fix broken fragment APIs 8 months ago

good first issue python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make a matrix multiplication implementation in Rust 8 months ago

enhancement good first issue help wanted python rust vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Provide better guidance for users creating indices 8 months ago

documentation good first issue help wanted python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

IVF_SQ index 8 months ago

enhancement help wanted rust vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Support `regexp_match` for `LargeUtf8` 8 months ago

good first issue help wanted python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Test which Arrow and Pandas datatypes are supported in write APIs 8 months ago

good first issue help wanted python ci arrow

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Allow users to set a distance threshold to consider two vectors a "very similiar" 8 months ago

enhancement good first issue help wanted python vector

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Index out of bounds using lance extension in duckdb v0.7.0 8 months ago

bug good first issue python duckdb rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make PyO3 functions as async function 8 months ago

enhancement good first issue python rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Make numpy optional pylance dependency 8 months ago

enhancement good first issue python

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Merge with more than a single column for `left_on` possible? 8 months ago

enhancement good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

added a `localstore` scheme in `object_store.rs` for testing 8 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Enable stricter clippy lints 8 months ago

enhancement good first issue help wanted rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feat: provide a Java SDK to read / write lance format 8 months ago

enhancement help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Fix torch related tests 8 months ago

bug good first issue help wanted python PyTorch

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

`load_indices` should include index config 8 months ago

enhancement good first issue help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

lance run benchmark panic 8 months ago

bug good first issue help wanted benchmark rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Android build fails due to `aarch64` in Rust 8 months ago

good first issue rust

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

V2 manifest path in Java 8 months ago

good first issue java

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Does lance support hdfs access? 8 months ago

enhancement help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

Consolidate the logic between ZoneMap and BloomFilters indexes 8 months ago

enhancement good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

The `InstrumentedRecordBatchStreamAdapter` is often misused and double counts CPU time 9 months ago

bug good first issue

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust

feat: bytes read and bytes write mectirc for per lance scanner and writer 9 months ago

help wanted

lance-format/lance

6.8K

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Rust

#apache-arrow#computer-vision#data-analysis#data-analytics#data-centric#data-format#data-science#dataops#deep-learning#duckdb#embeddings#llms#machine-learning#mlops#python#rust