What problem does it solve? Preparing high-quality training data from massive web scrapes like Common Crawl requires deduplication, quality filtering, and PII removal at terabyte scale, which is prohibitively slow and expensive on CPU-only pipelines. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU, processing 8TB in 7.5 hours on 8 A100 GPUs. - Quality Filtering & Classifiers: 30+ heuristic filters (word count, repeated lines, URL ratio) plus GPU classifiers for quality scoring and NSFW detection. - Multimodal Curation: Text, image (aesthetic/NSFW/CLIP), video (scene detection, embeddings), and audio (ASR, WER filtering) pipelines with RAPIDS and Dask scaling. - Use Case: Curate a Common Crawl dump by chaining language identification, heuristic filters, fuzzy deduplication, and PII redaction, then export clean Parquet files for LLM pretraining. ## Quick Start Use the nemo-curator skill to build a pipeline that loads my Parquet dataset, filters low-quality documents, removes fuzzy duplicates, and redacts PII before saving the curated output.
