Eventual-Inc/Daft

Add split_part string function about 11 hours ago

enhancement good first issue expression p2 (backlog)

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support ADBC reads and writes from databases 5 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support DataSources that wrap DataFrame templates 5 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Reader for WebDatasets File Format 6 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Writer for WebDatasets File Format 6 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Python Interface for IO extensions 6 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

temporal expressions parity with pyspark 6 days ago

good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Truncate IOConfig prints in plan outputs (`df.explain()`) 6 days ago

enhancement help wanted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`read_text` and `read_blob` functions 9 days ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support using OpenDAL Python library for additional filesystems 12 days ago

enhancement help wanted perf

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Migrate HudiScanOperator to DataSource 13 days ago

good first issue help wanted chore p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Migrate IcebergScanOperator to DataSource 13 days ago

good first issue help wanted chore p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Migrate DeltaLakeScanOperator to a DataSource 13 days ago

good first issue help wanted chore p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Mirror load_datasets api for read_huggingface 15 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Window function support 16 days ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

string expressions parity with pyspark 22 days ago

good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

functions decorated with `daft.func` don't respect the input type signature about 1 month ago

enhancement help wanted p2 (backlog) types

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

List-return UDF exceptions are swallowed as "Need at least 1 series to perform concat" about 1 month ago

bug help wanted p1 fix

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support upsert functionality in write_iceberg about 1 month ago

enhancement good first issue data-catalogs iceberg p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

math expressions parity with pyspark about 1 month ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`with_column`/`with_columns` improvements about 1 month ago

enhancement help wanted p2 (backlog) python refactor accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: Add `to_map` constructor function about 1 month ago

enhancement good first issue expression p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

perf(optimizer): eliminate sorts preceded by another sort about 1 month ago

enhancement help wanted p2 (backlog) perf

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Implement sample by size on ray runner about 2 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: Add top-level `daft.concat()` for concatenating multiple DataFrames about 2 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

chore: Bump minimum PyArrow version to 16 3 months ago

enhancement good first issue accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Parameterized queries to prevent SQL injection attacks 3 months ago

enhancement help wanted p2 (backlog) sql

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Integration test suites for catalog implementations 3 months ago

enhancement help wanted chore data-catalogs p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Fix round-trip write/read issues from Parquet/CSV/JSON 3 months ago

bug good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat: List flatten: List[List[T]] -> List[T] 3 months ago

enhancement good first issue

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

feat(dashboard): render Repartition spec in human-readable form instead of debug repr 3 months ago

enhancement good first issue p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for Arrow PyCapsule Interface 4 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Optimizer: common subplan elimination 4 months ago

help wanted p2 (backlog) perf accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

support writing csv without headers 4 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Remove calling functions to get Python DataTypes 4 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

fix: S3 HTTP client ignores HTTP_PROXY/HTTPS_PROXY environment variables 4 months ago

enhancement help wanted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`expr.list.map(...)` and `expr.list.filter(...)` 4 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[FEAT] User-defined aggregate functions 4 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Reading Daft-created partitioned Parquet with pandas fails 4 months ago

bug help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add support for `cast(DataType.duration(s/ms/us/ns)` 4 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

better error reporting for invalid sql statements 4 months ago

enhancement help wanted p2 (backlog) sql

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support point-in-time-join on a given set of dataframes 4 months ago

enhancement help wanted p1 accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Reduce total binary size by rewriting code 4 months ago

help wanted p1 accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

bug: coalesce does not exit early when it encounters a non-null value 4 months ago

bug help wanted p2 (backlog) perf accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

add `read_arrow` and `read_arrow_stream` methods 5 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for rolling joins and other special joins 5 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Missing support for Float16 type `FixedShapeTensorArray` or `ArrowTensorArray` from parquet/pyarrow sources 5 months ago

bug help wanted p2 (backlog) types accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add fp16 type support 5 months ago

enhancement help wanted p2 (backlog) types accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

fix the dev release always showing `0.30.0-dev` 5 months ago

bug help wanted build p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[PERF] Eliminate redundant HEAD requests when reading parquet 5 months ago

help wanted p2 (backlog) perf

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[SQL] Add SAMPLE LogicalPlan support 5 months ago

help wanted p2 (backlog) sql accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support selector expressions 5 months ago

enhancement help wanted p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

flatten nested error messages/better error handling 5 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`df.show` max_width does not work without manually setting the `format` option as well 5 months ago

bug good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add Kafka Data Source Support for Streaming Data Processing 5 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

add support for extended glob features, such as numeric ranges `{0..100}` 5 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support various image hash functions for deduplication. 5 months ago

enhancement good first issue help wanted p2 (backlog) types

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support version-hint.txt for reading iceberg tables 5 months ago

enhancement good first issue iceberg p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Terminal rendering of map types displays internal key-value struct 6 months ago

bug good first issue p2 (backlog) accepted

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`ddof` argument to `stddev` 6 months ago

enhancement good first issue help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add median function 6 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`daft.read_deltalake` returns a table when an invalid value for `version` is provided 6 months ago

bug good first issue data-catalogs p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support Dataframe cache/persist 6 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Assert Dataframe and schema for testing 6 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support `.as_T` cast methods where `T` is a Daft type with possible parameters 6 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add a `.list.contains()` expression 6 months ago

enhancement good first issue expression p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

time zone conversion features 6 months ago

enhancement good first issue help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Allowing GroupedDataFrame.agg_concat to also take in a delimiter like Expression.list.join 6 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Expression.var 6 months ago

AI Summary: This is a feature request to add `var()` (variance) support for expressions within Narwhals, specifically targeting Daft columns like `daft.col('a').var()`. The request also suggests including a `ddof` argument for controlling the degrees of freedom in the variance calculation.

Complexity: 3/5

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support specifying embedding dimensions for all providers 6 months ago

AI Summary: This feature request seeks to extend the `dimensions` parameter for text embeddings to all supported providers, as it currently only works for OpenAI. The goal is to implement dimension specification for other providers where feasible (e.g., using `truncate_dim` for `SentenceTransformer`) and to fail gracefully if a model does not support this functionality.

Complexity: 3/5

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

read_csv support `scan_tasks_min_size_bytes ` and `scan_tasks_max_size_bytes` to prevent single file too big 6 months ago

enhancement good first issue needs triage

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

custom quote characters when writing csv 6 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add support for Series[start:end] 6 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

custom delimiter when writing csv 6 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Explode returns null on empty or NULL lists 6 months ago

bug good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

custom date & timestamp formatting when writing csv 6 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

`uuid` function 7 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Allow overriding embedding dimension precision in OpenAI compatible embedding 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

daft emits DeprecationWarning on import 8 months ago

bug help wanted p1

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add support for writing `_SUCCESS` files in parquet operations, similar to Spark 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Make casting image to tensor easier 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

string casing functions 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

[EXPRESSIONS]: Distance functions 8 months ago

enhancement good first issue expression p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Hash rows of dataframe 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Missing mypy_boto3_glue in optional dependencies 8 months ago

bug good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

make it easier to enable different logging levels 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

chore: remove `HasMoreOutput(None)` from `StreamingSinkOutput` 8 months ago

enhancement good first issue p2 (backlog) needs triage

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Add examples for using sessions with providers and embed_text 8 months ago

documentation enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Daft doesn't work with OneLake 8 months ago

bug help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Expression.pow 8 months ago

enhancement good first issue help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Refactor the read/write lance API based on the Lance Rust SDK 8 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for `SHOW TABLES LIKE ...`for MemoryCatalog 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Apache Hudi Support Roadmap 8 months ago

enhancement help wanted p2 (backlog) feat

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

enh: DataFrame.join_asof 8 months ago

enhancement help wanted p2 (backlog) narwhals

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Merge support for DeltaLake 8 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support UUID data type 8 months ago

enhancement help wanted p2 (backlog) types

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Supporting fractional cpu count on the `ray` runner 8 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Comparison operators for list and struct types 8 months ago

enhancement good first issue help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Support for uint256/int256 8 months ago

enhancement help wanted p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Allow adding indices to explodes 8 months ago

enhancement good first issue p2 (backlog)

Eventual-Inc/Daft

5.6K

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Rust

#ai-engineering#ai-pipeline#arrow#artificial-intelligence#big-data#data-engineering#distributed#distributed-computing#distributed-systems#embeddings#etl#huggingface#iceberg#machine-learning#multimodal#parquet#python#ray#rust

Open Issues Need Help