skills.rest
richardnguyen0715
richardnguyen0715COMMUNITY
Explore all skills by this author →
GitHub

nemo-curator

GPU-accelerate LLM training data curation with fuzzy deduplication, quality filtering, and PII redaction.

Updated May 11, 2026
Search Tags:#GPU acceleration#deduplication#LLM training#quality filtering#data curation
One-click install
npx skills add https://github.com/richardnguyen0715/keep-it-real --skill nemo-curator-richardnguyen0715
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: nemo-curator
Source: https://github.com/richardnguyen0715/keep-it-real/tree/main/refer-projects/hermes-agent/optional-skills/mlops/nemo-curator
Command: npx skills add https://github.com/richardnguyen0715/keep-it-real --skill nemo-curator-richardnguyen0715

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nemo-curator, cudf, dask, rapids, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenges of preparing high-quality training datasets for large language models (LLMs), including data deduplication, quality filtering, and privacy redaction.

Core Features & Use Cases

  • GPU-accelerated Data Curation: Offers up to 16x faster fuzzy deduplication, 30+ quality filters, and PII redaction.
  • Multi-modal Support: Caters to text, image, video, and audio data.
  • Use Case: Ideal for cleaning web scraped data, deduplicating large corpora, or preparing datasets for LLM training.

Quick Start

Use the nemo-curator skill to filter and deduplicate your dataset 'llm_training_data.csv'.

Frequently Asked Questions about nemo-curator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate data curation and fuzzy deduplication for LLM training datasets?▼

GPU-accelerated data curation for LLM training speeds up fuzzy deduplication by up to 16x. It uses RAPIDS to scale across GPUs, efficiently cleaning and processing large text corpora.

Can I use GPU acceleration to filter and redact PII from web scraped data?▼

Yes, you can redact PII and apply over 30 quality filters to web scraped data using GPU acceleration. This ensures privacy and dataset quality for LLM training.

Does GPU-accelerated data curation support multi-modal formats like image, video, and audio?▼

GPU-accelerated data curation supports multi-modal data including text, image, video, and audio formats. It processes these diverse datasets for comprehensive LLM training preparation.

Do I need specific libraries like RAPIDS and Dask to run GPU-accelerated data curation?▼

Yes, you need nemo-curator, cudf, dask, and rapids libraries installed. These dependencies provide the necessary GPU acceleration and distributed computing environment for data curation.

Related Skills

Explore More Data & Analytics Skills→
KetomihineKetomihine
1

rapids-singlecell-complete

Accelerate single-cell RNA-seq preprocessing and clustering on AnnData with RAPIDS.

Community
Advanced
sheawinklersheawinkler
78

nemo-curator

Deduplicate, filter, and redact multi-modal training data with GPU acceleration.

Community
Advanced
blueskies1818blueskies1818
1

nemo-curator

Accelerates data curation for LLM training with GPU-based deduplication, filtering, and PII redaction.

Community
Advanced
gqf2008gqf2008

nemo-curator

Automate GPU-accelerated deduplication, quality filtering, and PII redaction for large training corpora.

Community
Advanced