Hugging Face ships tokenizers v1 — encoding up to 30x faster
The plumbing of AI got two small releases today — one that stops GPUs from idling on CPU work, one that puts image generation on a laptop-grade footprint.
Hugging Face shipped the first major version of its tokenizers library, and the headline number is real: 3 to 30 times faster encoding than v0.23 on a single thread. The tokenizer is the step that turns text into the integer IDs a model reads, and it runs in four stages — normalization, pre-tokenization, the model stage that maps pre-tokens to vocabulary IDs, and post-processing. v1 keeps the output identical to v0.23: same token IDs, same API, same vocabulary and merge ranks. Every change is in the encoding path underneath. The regular expression that splits input into pre-tokens is now a hand-written function using CPU SIMD instructions, deciding 64 bytes per register operation instead of scanning character by character. A thread-local cache stores the token IDs for a repeated word after the first merge, so later occurrences skip the merge loop entirely. That loop itself no longer touches the allocator — the working set lives in a caller-owned scratch buffer, and a batch of pre-tokens is processed in one model call rather than one per pre-token.
The benchmark detail matters as much as the speedup. The measured range across ten model families runs from 3x on t5-base to 30x on GPT-2, and the gains depend on whether the model's split pattern is one of the handful of grammars the fast path recognizes — a tokenizer outside that set keeps the regex engine and none of the speed-up, which is exactly why the spread is so wide. Multi-thread scaling hits 76% of linear across eight workers, with each thread drawing its own scratch buffer and cache so they no longer queue on one lock. Why it matters: the tokenizer was never the bottleneck until it was. Training on massive datasets, serving many concurrent requests, or repeatedly processing long agentic inputs can push enough text through the CPU that the model starves waiting for data — the failure mode is an expensive GPU sitting idle. IBM, NVIDIA and the ExecuTorch team contributed patches across hardware. Hugging Face also credits the wider open-source field — gigatoken, tiktoken-rs, tokie and others — for ideas that reached the rewrite. If tokens are still a fuzzy concept, our explainer on what a token actually is covers the unit the whole stack is denominated in.
SupraLabs released Supra2-IMG, a 105-million-parameter diffusion transformer that generates images at 256×256 in about 2 seconds on a GPU and roughly 20 seconds on a CPU. The team trained it entirely from scratch in nine hours on a single H100 80GB pod rented from Runpod, data preparation included — a budget in the tens of dollars, not the millions that usually sit behind an image model. The samples shipped with the post use one fixed setting (seed 0, 50 steps, guidance 3.0) and the author states they are not cherry-picked. That claim drew the expected pushback: one commenter pointed out that "state of the art" deserves an asterisk, since the images are good for 105M parameters but not competitive with frontier generators, and the author's defense is the honest version — SOTA for the size class, not overall. The response from the local-model crowd was more telling than the skepticism: people are used to text-to-image meaning enormous checkpoints and datacenter GPUs, and a model that fits anywhere and runs on a CPU feels like a different category. SupraLabs says a 1024×1024 version is already in progress, and the inference script plus weights are on Hugging Face.
What to watch: whether the remaining tokenizer model families move onto the new merge loop before 1.0, and whether Supra2-IMG's 1024×1024 build keeps its single-GPU training budget — the cheap end of image generation is where the interesting constraints live.
Does a 105M-parameter image model have a real use case outside research, or is small-for-its-size a benchmark story? Tell us in the comments.
Sources: Hugging Face — tokenizers v1: encode, decode and scaling, measured · tokenizers v1 release candidate (crates.io) · huggingface/tokenizers PR #2365 (GitHub) · Hacker News / r/LocalLLaMA — tokenizers v1 (Rust) · Supra2-IMG model card (Hugging Face) · r/LocalLLaMA — Supra2-IMG release thread
