panodata

panodata/omniload

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

0 stars 2 forks 0 watchers Python MIT License

data-engineering data-ingestion data-lake data-load data-loaders data-loading data-loading-and-analysis data-loading-and-cleaning data-loading-framework data-pipeline data-warehouse dataloader dlt dlt-destination etl etl-framework warehouse warehouse-management

View on GitHub Website

29 Open Issues Need Help Last updated: Jul 3, 2026

Open Issues Need Help

View All on GitHub

Write support for XLSX about 6 hours ago

help wanted data format

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

PyStreamXL about 6 hours ago

help wanted data format

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

The new canonical `file://` destination 3 days ago

help wanted connector refinement needs discussion

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Meilisearch 3 days ago

help wanted connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

ADBC 7 days ago

help wanted pitch connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Parquet export writes an integer the Parquet reader cannot read back 8 days ago

help wanted refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: HTTP `block_size=0` yields a handle the seeking readers cannot use 8 days ago

help wanted refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Make `dlt-filesystem` a dedicated package 9 days ago

help wanted needs discussion

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Give the format writers more love 9 days ago

help wanted data format

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Feather 15 days ago

help wanted refinement data format

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Avro 15 days ago

help wanted refinement data format

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Unity Catalog / Delta Lake integration test 18 days ago

help wanted quality

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Tiger Data 23 days ago

help wanted connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Migrate remote filesystem adapters from fsspec to pyarrow.fs (Azure Blob Storage, GCS, HDFS, S3) about 1 month ago

help wanted performance

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Migrate from `HttpSource` to `fsspec.HTTPFileSystem` about 1 month ago

help wanted refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Clarify `filesystem_incremental` option ownership in fsspec sources about 2 months ago

help wanted question

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Database: Support 3-component table names (`catalog.schema.table` aka. `project.dataset.table`) about 2 months ago

help wanted refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Database: Load file-based databases from remote filesystems about 2 months ago

help wanted connector refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

[Deployment] Consume managed secrets about 2 months ago

help wanted readiness

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Integration tests for remote filesystems 2 months ago

help wanted quality

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Object Storage Service (OSS) `oss://` 2 months ago

help wanted connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Incremental loading for filesystem resources 2 months ago

help wanted question refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Migrate from `fsspec.LocalFileSystem` to `pyarrow.fs.LocalFileSystem` 2 months ago

help wanted performance

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Read _multiple_ spreadsheets from workbook files 2 months ago

help wanted refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Document current incremental loading mechanism 3 months ago

documentation help wanted refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Azure Data Lake Storage Gen2 `adls://` 3 months ago

help wanted connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Azure Blob Storage `az://` 3 months ago

help wanted connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

Filesystem: Named hint parameters for URIs 3 months ago

help wanted connector refinement

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management

ValueError: Unsupported destination scheme: file 3 months ago

enhancement help wanted connector

panodata/omniload

0

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

Python

#data-engineering#data-ingestion#data-lake#data-load#data-loaders#data-loading#data-loading-and-analysis#data-loading-and-cleaning#data-loading-framework#data-pipeline#data-warehouse#dataloader#dlt#dlt-destination#etl#etl-framework#warehouse#warehouse-management