Skills · davila7

Tokenization SentencePiece: language-independent tokenizer skill

A skill for a language-independent tokenizer that treats text as raw Unicode and supports BPE and Unigram algorithms.

This text was machine-translated from the publisher's own information and has not been checked by a human editor. The publisher's original text is the reference.

This resource was imported automatically from the sources listed below and has not been reviewed by the AI Gate team yet. The description is the publisher's own, as published. Check the repository before installing.

What it does

This skill covers a tokenizer that treats text as raw Unicode, so it does not depend on language-specific preprocessing. It supports the BPE and Unigram algorithms and can train directly on raw text without pre-tokenization. The vocabulary it produces is deterministic.

The tokenizer suits multilingual work, including CJK languages, and cases where reproducible tokenization matters. Models such as T5, ALBERT, XLNet and mBART are listed as using it.

The publisher reports throughput of about 50k sentences per second and memory use of about 6MB. The licence is MIT. The source gives no setup steps or software requirements.

Safety

Our team has not reviewed this resource. Only automated rules checked it; resources with high-risk signals are not published. These are signals, not a guarantee that installing it is safe. Try it in a sandbox project first.

  • Runs code or changes an assistant's behaviour

Sources

  • AI Templates (aitmpl) first seen 13 Nov 2026 · last seen 17 Nov 2026 · its page on the source · figures reported by the source: aitmpl downloads 27
    tokenization-sentencepiece

    Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

    v1.0.0
    As shown on www.aitmpl.com when we last read it
Not reviewedSkills
github

Onboard context matic

Onboard context matic is a skill published by GitHub and hosted in its awesome-copilot repository.

Software development
Updated 17 Nov 2026
Not reviewedSkills
google

Bigquery migration validator

A skill published by Google for validating BigQuery migrations.

Data & databases
Updated 17 Nov 2026
Not reviewedSkills
github

Openapi to application code

A GitHub skill whose name points to turning OpenAPI definitions into application code; no further details are available.

Software development
Updated 17 Nov 2026
Not reviewedSkills
openai

Twilio notifications alerts advisor

A development skill from openai that offers advice on notifications and alerts using Twilio.

Software development
Updated 14 Nov 2026