What it does
The skill covers data curation for LLM training across text, image, video and audio. Its listed features are fuzzy deduplication, which the publisher describes as 16× faster, quality filtering with 30+ heuristics, semantic deduplication, PII redaction and NSFW detection. Processing scales across GPUs using RAPIDS.
It is intended for preparing high-quality training datasets, cleaning web data and deduplicating large corpora. Someone assembling a training set from raw web or multimedia sources would reach for it to filter, deduplicate and redact data before training.
Running it involves GPU hardware and the RAPIDS stack. The resource is released under the MIT license.
Limitations and requirements
Requires GPUs and RAPIDS, as stated in the publisher's description.
Safety
Our team has not reviewed this resource. Only automated rules checked it; resources with high-risk signals are not published. These are signals, not a guarantee that installing it is safe. Try it in a sandbox project first.
- Runs code or changes an assistant's behaviour
Sources
- AI Templates (aitmpl) first seen 13 Nov 2026 · last seen 18 Nov 2026 · its page on the source · figures reported by the source: aitmpl downloads 133
data-processing-nemo-curator
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
v1.0.0As shown on www.aitmpl.com when we last read it