skills.rest
80portisfound
80portisfoundCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, quality filtering, and PII redaction.

Updated May 3, 2026
Search Tags:#deduplication#quality filtering#data curation#pii redaction#gpu acceleration#nemo curator#llm training data
One-click install
npx skills add https://github.com/80portisfound/vibe-learning --skill nemo-curator-80portisfound
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/80portisfound/vibe-learning/tree/main/packages/hermes/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/80portisfound/vibe-learning --skill nemo-curator-80portisfound

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve? Preparing high-quality training data from large web scrapes or multi-modal corpora is slow and expensive on CPUs, and raw datasets contain duplicates, low-quality content, PII, and NSFW material that degrade model training. ## Core Features & Use Cases - Deduplication at scale: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster on GPUs, processing 8TB of RedPajama v2 in 7.5 hours. - Quality and safety filtering: 30+ heuristic filters (word count, repeated lines, URL ratio) plus classifier-based quality and NSFW scoring, and PII redaction for emails, phone numbers, and names. - Multi-modal curation: Image aesthetic/NSFW filtering with CLIP embeddings, video scene detection and clip extraction, and audio ASR transcription with WER filtering. - Use Case: Curate a Common Crawl dump by chaining language identification, heuristic filters, fuzzy deduplication, and PII redaction, then save the cleaned corpus to Parquet across an 8-GPU Dask cluster. ## Quick Start Use the nemo-curator skill to build a GPU pipeline that filters, deduplicates, and redacts PII from my Common Crawl parquet files.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate a large text dataset for LLM training?▼

Use NeMo Curator's FuzzyDuplicates module with MinHash and LSH to remove near-duplicates at scale. On GPUs it processes 8TB in about 7.5 hours, roughly 16× faster than CPU-based deduplication.

What is the difference between exact, fuzzy, and semantic deduplication?▼

Exact deduplication removes identical documents via hashing, fuzzy deduplication catches near-duplicates using MinHash and LSH, and semantic deduplication uses embedding similarity to detect paraphrases. Fuzzy deduplication offers the best speed and recall trade-off for large corpora.

Does NeMo Curator work without a GPU?▼

Yes, a CPU-only installation is available via the cpu extra, but it is significantly slower. GPU acceleration with RAPIDS delivers 10-16× speedups for deduplication and filtering workloads.

Can NeMo Curator process images, video, and audio?▼

Yes, it supports multi-modal curation including image aesthetic and NSFW filtering with CLIP embeddings, video scene detection and clip extraction, and audio ASR transcription with WER-based filtering.

How do I remove PII from training data?▼

Use the PIIRedactor modifier with the Modify module to detect and replace or redact entities such as email addresses, phone numbers, person names, and locations across your dataset.

When should I use datatrove or dolma instead of NeMo Curator?▼

Choose datatrove or dolma when you lack GPU infrastructure or need a CPU-based open-source pipeline. NeMo Curator is better suited when processing terabyte-scale corpora where GPU acceleration reduces time and cost substantially.

Related Skills

Explore More Data & Analytics Skills→
NousResearchNousResearch
239.1k

nemo-curator

Curate LLM training data via GPU-accelerated deduplication, filtering, and PII redaction.

Official
Advanced
inbrainfuninbrainfun
174

skill-optimizer

Optimize existing skills against benchmarks using validation-gated mutations.

Community
Advanced
Veloxia-agencyVeloxia-agency

data-quality-auditor

Audit CSV datasets for completeness, consistency, validity, uniqueness, and timeliness.

Community
Advanced
MesferAliMesferAli

ray-data

Processes large-scale ML datasets with distributed streaming across CPUs and GPUs.

Community
Advanced