GigaToken, an open-source Rust tokenizer, claims up to 1,353x faster throughput than Hugging Face

Marcel Roed released GigaToken, an open-source tokenizer for language models written in Rust, whose own benchmarks claim throughput up to 1,353x faster than Hugging Face’s widely used tokenizers library, according to the project.

A tokenizer converts raw text into the tokens a model reads and generates; it runs before every training and inference step, so its speed sets a floor on how fast large datasets can be processed. GigaToken is offered as a drop-in replacement for Hugging Face’s library under an MIT license and is installable via pip.

The project’s benchmarks claim roughly 280x to 1,353x faster throughput than Hugging Face’s tokenizers and up to 681x faster than OpenAI’s tiktoken, measured across tokenizer definitions for Llama, Qwen, DeepSeek, Gemma and GPT-2 on three CPU architectures. On an AMD EPYC 9565 with 144 cores, GigaToken tokenizes GPT-2-style text at 24.53 gigabytes per second versus 24.8 megabytes per second for Hugging Face’s tokenizer, per those benchmarks; on an Apple M4 Max it reaches 8.79 gigabytes per second against 6.9 megabytes per second. Roed says the throughput would let a single machine tokenize a Common Crawl-scale dataset in under 6.5 hours.

The figures are the author’s own, run on his chosen hardware and inputs, with no independent technical review yet. Tokenizer benchmarks are sensitive to configuration, and the enormous multipliers, thousands of times faster, invite scrutiny over whether the comparison holds across real workloads and non-batched inputs.

Still, the release drew wide attention among developers, a sign of appetite for faster data-processing tooling as training runs grow. Independent benchmarks on varied hardware will show whether GigaToken’s numbers survive contact with other people’s pipelines.