nemo-curator

What problem does it solve?

This Skill streamlines the creation of high-quality datasets for training Large Language Models by accelerating data cleaning, deduplication, and filtering processes using GPU power.

Core Features & Use Cases

• Fast Deduplication: Utilizes fuzzy and semantic deduplication techniques that are significantly faster on GPUs.
• Quality Filtering: Employs over 30 heuristics and classifiers to remove low-quality, toxic, or redundant data.
• Multimodal Support: Capable of curating text, image, video, and audio data.
• PII Redaction: Automatically identifies and redacts personally identifiable information.
• Use Case: Prepare a massive web scrape dataset for LLM training by removing duplicate documents, filtering out low-quality content, and redacting sensitive information, all in a fraction of the time it would take on CPUs.

Quick Start

Use the nemo-curator skill to prepare a dataset for LLM training by applying quality filters and deduplication.