nemo-curator_skill

Overview

This skill provides GPU-accelerated data curation tools for preparing high-quality training datasets for LLMs and multi-modal models. It supports text, image, video, and audio pipelines with fuzzy/semantic deduplication, 30+ heuristic quality filters, PII redaction, and NSFW detection. The implementation scales across GPUs (Dask/CUDA) for large corpora and delivers large speedups and lower TCO versus CPU-only workflows.

How this skill works

The skill runs staged pipelines: heuristic quality filtering, exact/fuzzy/semantic deduplication, PII redaction, and classifier-based filtering. GPU kernels accelerate MinHash/LSH fuzzy deduplication, embedding-based semantic deduplication, and batched classifier inference. Pipelines operate on Parquet/JSONL/CSV inputs, integrate with Dask CUDA for multi-GPU scaling, and export curated Parquet or JSONL outputs.

When to use it

• Curating web-scraped corpora (Common Crawl, RedPajama) before model training
• Removing duplicates from multi-terabyte text collections with GPU speedups
• Preparing multi-modal datasets (images, video, audio) with NSFW and quality filters
• Redacting PII and enforcing safety policies at dataset scale
• Reducing compute and cost for large-scale deduplication and filtering tasks

Best practices

• Run quality heuristics first (word counts, URL ratio, repeated lines) to cheaply remove noise
• Use exact deduplication before fuzzy/semantic passes to reduce candidate pairs
• Tune MinHash/LSH parameters (num_hashes, num_buckets) to balance recall and speed
• Embed-based semantic deduplication for paraphrase-heavy corpora; choose a lightweight embedder for scale
• Leverage Dask CUDA/LocalCUDACluster for near-linear multi-GPU scaling and lower TCO

Example use cases

• Curating Common Crawl slices: filter language, quality heuristics, deduplicate, redact PII, save Parquet
• Assembling an image dataset: aesthetic scoring, NSFW filtering, CLIP embedding export
• Video dataset pipeline: scene detection, clip extraction, video embeddings for retrieval
• Audio transcription cleanup: ASR inference, WER filtering, duration constraints before training speech models
• Large-scale fuzzy deduplication of RedPajama-style corpora using A100 or multi-GPU clusters