skills.rest
azaanaliraza
azaanalirazaCOMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Curate and deduplicate LLM training datasets on GPUs with NeMo Curator.

Updated Apr 18, 2026
Search Tags:#deduplication#quality filtering#data curation#pii redaction#gpu acceleration#nemo curator#llm training data
One-click install
npx skills add https://github.com/azaanaliraza/operarius --skill nemo-curator-azaanaliraza
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/azaanaliraza/operarius/tree/main/src-tauri/bin/hermes/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/azaanaliraza/operarius --skill nemo-curator-azaanaliraza

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve? Preparing high-quality training data from massive raw corpora like Common Crawl is slow and expensive on CPUs, and low-quality, duplicated, or unsafe content degrades LLM training outcomes. ## Core Features & Use Cases - GPU-Accelerated Deduplication: Exact, fuzzy (MinHash + LSH), and semantic deduplication running up to 16× faster than CPU pipelines. - Quality Filtering & Safety: 30+ heuristic filters, quality classifiers, NSFW detection, and PII redaction for text, image, video, and audio datasets. - Multi-GPU Scaling: Distributed processing across GPU clusters using RAPIDS, cuDF, and Dask for terabyte-scale corpora. - Use Case: Curate an 8TB Common Crawl scrape by filtering low-quality documents, removing near-duplicates, redacting PII, and exporting clean Parquet files for LLM pretraining. ## Quick Start Ask the AI to build a NeMo Curator pipeline that filters, deduplicates, and redacts PII from a Parquet dataset of web-scraped text.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deduplicate a large text dataset for LLM training?▼

Use NeMo Curator's FuzzyDuplicates module, which applies MinHash and LSH to remove near-duplicates. On GPUs it processes 8TB in about 7.5 hours, roughly 16× faster than CPU-based deduplication.

What is the difference between exact, fuzzy, and semantic deduplication?▼

Exact deduplication removes identical documents via hashing, fuzzy deduplication catches near-duplicates using MinHash and LSH, and semantic deduplication uses embedding similarity to detect paraphrases. Fuzzy dedup offers the best speed-to-recall trade-off for large corpora.

Does NeMo Curator work without a GPU?▼

Yes, a CPU-only installation is available via pip install nemo-curator[cpu], but it runs significantly slower. GPU acceleration with RAPIDS delivers 10-16× speedups on deduplication and filtering workloads.

Can NeMo Curator process image and video datasets?▼

Yes, it supports multimodal curation including aesthetic scoring and NSFW filtering for images, scene detection and clip extraction for video, and ASR transcription with WER filtering for audio.

NeMo Curator vs datatrove for data curation?▼

NeMo Curator is GPU-accelerated and optimized for large-scale deduplication and multimodal curation, while datatrove is CPU-based and lighter weight. Choose NeMo Curator when processing terabyte-scale corpora with available GPU infrastructure.

Related Skills

Explore More Data & Analytics Skills→
KarlinskySKarlinskyS

nemo-curator

Deduplicate, filter, redact PII, and remove NSFW content from large multimodal corpora.

Community
Advanced
davila7davila7
30.5k

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Community
Advanced
jleechanorgjleechanorg

nemo-curator

Automate GPU-accelerated filtering, deduplication, PII redaction, and NSFW detection on multimodal datasets.

Official
Advanced
granietgraniet
247

llama-cpp

Run large language model inference on non-NVIDIA hardware with llama.cpp.

Community
Intermediate