nemo-curator

Use NeMo Curator's FuzzyDuplicates module with MinHash and LSH to remove near-duplicates at scale. On GPUs it processes 8TB in about 7.5 hours versus 120 hours on CPU, and you can tune num_hashes, num_buckets, and the Jaccard threshold for recall.

NeMo Curator is GPU-accelerated and best for very large datasets where speed matters, offering 16× faster fuzzy deduplication. datatrove and dolma are CPU-based open-source alternatives suited for smaller workloads or environments without GPUs.

Yes, NeMo Curator supports multimodal curation including image aesthetic scoring, NSFW detection, and CLIP embeddings, plus video scene detection, clip extraction, and InternVideo2 embeddings. Audio curation includes ASR transcription and WER-based filtering.

Yes, install the CPU-only extra with nemo-curator[cpu], but processing is significantly slower. GPU acceleration via RAPIDS delivers 10-16× speedups for deduplication and filtering, so CPU mode suits only small datasets.

Initialize a Dask CUDA cluster with LocalCUDACluster or get_client(cluster_type="gpu", n_workers=8), then run pipeline stages normally. NeMo Curator scales near-linearly across GPU nodes for operations like fuzzy deduplication.

Fuzzy deduplication achieves roughly 95% recall and only catches near-duplicates, not paraphrases or rewrites. For semantically similar content, use SemanticDuplicates with embedding models, though it is slower and more compute-intensive.