skills.rest
sheawinkler
sheawinklerCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

78|16|Updated Apr 23, 2026
Search Tags:#multimodal#gpu#deduplication#pii-redaction#data-curation#nsfw-detection
One-click install
npx skills add https://github.com/sheawinkler/hermes-agent-ultra --skill nemo-curator-sheawinkler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/sheawinkler/hermes-agent-ultra/tree/main/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/sheawinkler/hermes-agent-ultra --skill nemo-curator-sheawinkler

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

NeMo Curator accelerates the preparation of high-quality training data for multi-modal LLMs by combining GPU-accelerated deduplication, quality filtering, and redaction into a single workflow.

Core Features & Use Cases

  • GPU-accelerated deduplication (exact, fuzzy, semantic) across text, images, and other modalities.
  • 30+ quality filters to prune low-quality or unsafe data.
  • PII redaction and NSFW detection to sanitize training corpora.
  • Scales across GPU clusters with RAPIDS for large datasets and multi-modal curation.
  • Use cases include preparing RedPajama-type datasets, cleaning web data, and deduplicating large corpora for LLM training.

Quick Start

Run Nemo Curator on a GPU cluster to deduplicate, filter quality, redact PII, and curate multi-modal training data.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prepare large-scale training data for multi-modal LLMs?▼

Prepare large-scale training data for multi-modal LLMs by using GPU-accelerated data curation to deduplicate, filter, and redact text, image, video, and audio datasets. This ensures high-quality inputs for model training.

What is GPU-accelerated semantic deduplication and how does it work for training corpora?▼

GPU-accelerated semantic deduplication identifies and removes semantically similar data points in training corpora. It scales across GPU clusters using RAPIDS to efficiently clean large-scale text and multi-modal datasets.

Do I need the RAPIDS stack to run GPU-accelerated data curation?▼

Yes, you need the RAPIDS stack installed on a Linux or macOS environment to run GPU-accelerated data curation. It provides the necessary GPU compute backend for scaling deduplication and filtering tasks.

Can I redact PII and detect NSFW content in large web datasets?▼

Yes, you can redact PII and detect NSFW content in large web datasets. The data curation pipeline includes specific modules to sanitize training corpora and prune low-quality or unsafe data.

What is the best way to deduplicate large text corpora for LLM training?▼

The best way to deduplicate large text corpora is using a GPU-accelerated pipeline that performs exact, fuzzy, and semantic deduplication. This method scales across GPU clusters to clean RedPajama-type datasets.

Related Skills

Explore More Data & Analytics Skills→
zli5460zli5460
2

nemo-curator

Remove duplicates and filter low-quality data from large-scale datasets.

Community
Advanced
pmcdowallpmcdowall

fine-tuning-with-trl

Align large language models via reinforcement learning and preference-based fine-tuning workflows.

Community
Advanced
AlexiosBluffMaraAlexiosBluffMara
2

nemo-curator

Curate multi-modal LLM training data with GPU-accelerated deduplication and filtering.

Community
Advanced
kaminocorpkaminocorp
1

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Official
Advanced