skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

1|Updated Mar 12, 2026
Search Tags:#gpu#deduplication#quality filtering#data curation#nvidia#pii redaction#llm training data
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill nemo-curator-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/kaminocorp/hermes-alpha-hunter/tree/main/skills/mlops/evaluation/nemo-curator
Command: npx skills add https://github.com/kaminocorp/hermes-alpha-hunter --skill nemo-curator-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve? Preparing high-quality training data from massive web scrapes and raw corpora is slow and expensive on CPU, and low-quality or duplicate content degrades LLM training outcomes. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU on multi-terabyte datasets. - Quality Filtering & Safety: 30+ heuristic filters, quality classifiers, NSFW detection, and PII redaction for text, image, video, and audio data. - Use Case: Curate a Common Crawl dump by filtering short or repetitive documents, removing near-duplicates with fuzzy dedup across 8 GPUs, redacting emails and phone numbers, and exporting clean Parquet files for LLM pretraining. ## Quick Start Use the nemo-curator skill to deduplicate and quality-filter my text dataset stored in Parquet files using GPU acceleration.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate a large text dataset with NeMo Curator?▼

Use ExactDuplicates for identical documents, FuzzyDuplicates with MinHash and LSH for near-duplicates, or SemanticDuplicates with embedding models for paraphrases. Fuzzy deduplication on GPU processes 8TB in about 7.5 hours versus 120 hours on CPU.

NeMo Curator vs datatrove for data curation?▼

NeMo Curator targets GPU-accelerated curation with 16× faster fuzzy deduplication and multi-modal support for images, video, and audio. Datatrove is CPU-based and open-source, making it suitable when GPU infrastructure is unavailable.

Does NeMo Curator support image and video datasets?▼

Yes, it includes image curation with aesthetic scoring, NSFW filtering, and CLIP embeddings, plus video curation with scene detection, clip extraction, and InternVideo2 embeddings. Audio curation supports ASR transcription and WER-based filtering.

Can NeMo Curator run without a GPU?▼

A CPU-only installation is available via the cpu extra, but it runs significantly slower. GPU acceleration with RAPIDS delivers 10-16× speedups for deduplication and filtering, so GPUs are recommended for large datasets.

How do I scale NeMo Curator across multiple GPUs?▼

Initialize a Dask CUDA cluster with LocalCUDACluster or get_client with cluster_type set to gpu and the desired worker count. Processing scales near-linearly across GPU nodes for operations like fuzzy deduplication.

Related Skills

Explore More Data & Analytics Skills→
tadod12tadod12

nemo-curator

Automate multimodal dataset curation with deduplication, quality filtering, PII redaction, and NSFW detection.

Community
Advanced
K-Dense-AIK-Dense-AI
41.1k

waypoint-bio

Embed, fine-tune, and benchmark microbiome taxonomic profiles with Outpost Bio's Waypoint foundation models.

Official
Advanced
AissenLiuAissenLiu

nemo-curator

Automate GPU-accelerated data curation workflows for large multimodal datasets.

Community
Advanced
MesferAliMesferAli

ray-data

Processes large-scale ML datasets with distributed streaming across CPUs and GPUs.

Community
Advanced