skills.rest
ZardLi1115
ZardLi1115COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

Filter noisy multimodal datasets and remove exact, fuzzy, and semantic duplicates.

4|Updated May 18, 2026
Search Tags:#multimodal#data curation#rapids#pii redaction#llm training data#nsfw filtering#gpu deduplication
One-click install
npx skills add https://github.com/ZardLi1115/zedclaw --skill nemo-curator-zardli1115
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/ZardLi1115/zedclaw/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/ZardLi1115/zedclaw --skill nemo-curator-zardli1115

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes references (resource) components.

What problem does it solve?

NeMo Curator reduces noisy, duplicated, unsafe, and low-quality content so your LLM training sets stay clean, diverse, and privacy-safe without spending weeks manually auditing web-scale corpora.

Core Features & Use Cases

  • GPU-accelerated multimodal curation: Apply quality filtering and deduplication across text, images, video, and audio at scale.
  • Fast deduplication pipelines: Use exact, fuzzy (MinHash+LSH), and semantic (embedding-based) deduplication to remove repeats and near-repeats.
  • Safety and privacy controls: Perform PII redaction and NSFW detection/classifier filtering to reduce sensitive or harmful content.

Quick Start

Install nemo-curator with GPU support and run a pipeline that filters for quality, removes fuzzy duplicates, redacts PII, and writes the curated output to Parquet for training ingestion.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I curate multimodal LLM training data to remove duplicates and unsafe content?▼

You can curate multimodal LLM training data using GPU-accelerated pipelines that filter low-quality content, remove exact and fuzzy duplicates, and apply PII redaction with NSFW filtering for privacy-safe datasets.

What is GPU deduplication and how does it work for large text corpora?▼

GPU deduplication processes large text corpora using RAPIDS, applying exact, fuzzy (MinHash+LSH), and semantic embedding-based methods to remove repeats and near-repeats across multiple GPUs.

Do I need RAPIDS and multiple GPUs to run nemo-curator for data preparation?▼

Yes, nemo-curator requires GPU-accelerated processing with RAPIDS across multiple GPUs to handle web-scale multimodal corpora curation efficiently.

How do I redact PII and filter NSFW content from web-scale datasets?▼

Redact PII and filter NSFW content from web-scale datasets by applying built-in safety and privacy controls that detect and redact sensitive information while filtering harmful multimodal content.

What's the best way to prepare noisy web data for downstream model training?▼

The best way to prepare noisy web data for model training is running a curation pipeline that filters low-quality content, removes duplicates, redacts PII, and writes the curated output to Parquet for training ingestion.

Does nemo-curator support curation for image, video, and audio data?▼

Yes, nemo-curator supports multimodal curation across text, images, video, and audio, applying quality heuristics and safety controls to prepare diverse web-scale corpora for training.

Related Skills

Explore More Data & Analytics Skills→
KappTech88KappTech88

nemo-curator

Curate multimodal datasets for LLM training with GPU-accelerated pipelines.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

nemo-curator

Curate multimodal datasets with quality filtering, deduplication, and PII redaction.

Community
Advanced
markwang2658markwang2658
31

nemo-curator

Automates GPU-accelerated data curation for LLM training datasets with deduplication and quality filtering.

Community
Advanced
jleechanorgjleechanorg

nemo-curator

Automate GPU-accelerated filtering, deduplication, PII redaction, and NSFW detection on multimodal datasets.

Official
Advanced