NVIDIA-NeMo

NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

1.7K stars 296 forks 1.7K watchers Python Apache License 2.0

data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication

31 Open Issues Need Help Last updated: Jul 9, 2026

Open Issues Need Help

View All on GitHub

Support GlotLID for language/script identification. 8 days ago

good first issue Feature Text

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

[Ray] Pass expected language into `FastTextLangId` filter 26 days ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Text readers hang instead of failing fast when using `RayDataExecutor` 28 days ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Add support for data-designer 0.5.0 2 months ago

enhancement good first issue community-request waiting-on-maintainers

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Add a benchmark for Cloud I/O for text 3 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Alternateive to nvcv_image 3 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

See if all workflows expose file_extensions for reader i/o 3 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Bug: Overly permissive url regex 3 months ago

bug good first issue community-request waiting-on-maintainers

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Automatically detect when `IS_FANOUT_STAGE` should be set to True 3 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Ray Data Adapter (backend) should use compute instead of concurrency due to deprecation 4 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Default jsonl_writer to write data with kwarg force_ascii=False 4 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Audio stages: most fan-out stages do not set RayStageSpecKeys.IS_FANOUT_STAGE 5 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

RayDataStageActorAdapter never calls teardown() on stages 5 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Better error messages when reading large parquet files 5 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Update stage name for nemotron-cc SDG stage 5 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

RayDataExecutor: start GPU actor pools at full concurrency instead of scaling up one-by-one 5 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Video vLLM stages have race condition during setup due to torch.compile cache 6 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Add benchmarking script for FastText filters 6 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Documentation 'Installation" & "Configuration" links broken 6 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Unable to use models which need `trust_remote_code=True` in `EmbeddingCreatorStage` 6 months ago

bug good first issue community-request

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Make sure all reserved keywords are attributes and not properties [name / resources / batch_size] 6 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Audio stage names aren't being propogated as noticed in benchmark script 6 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Reorganize directory structure for `DocumentFilter` and `DocumentModifier` 6 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Support batchsize/process_batch in the minHash stage 6 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Keeping imports consistent across modalities, for images 6 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Consider removing MegatronTokenizerWriter from writer/__init__.py 7 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Improve error handling for `blocksize` / `files_per_partition` which can result in error: `Invalid file path or buffer object type: <class 'tuple'>` 7 months ago

bug good first issue community-request

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Standardize ID naming for deduplication 7 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Warn if n_clusters < 1000 as a default in SemDedup 7 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Enhance YAML support in Curator 7 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Rename `pred_column` and `prob_column` to fields 10 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication