https://www.msuiche.com/tags/pyo3/Recent content in PyO3 on Matt SuicheHugoen-usWed, 15 Oct 2025 00:00:00 +0000https://www.msuiche.com/posts/rustbpe-high-performance-bpe-tokenizer-training-in-rust/Wed, 15 Oct 2025 00:00:00 +0000https://www.msuiche.com/posts/rustbpe-high-performance-bpe-tokenizer-training-in-rust/<h2 id="introduction">Introduction<a href="#introduction" class="anchor" aria-label="Link to Introduction">#</a></h2> <p>Byte Pair Encoding (BPE) tokenization is used in modern language models, but efficient training implementations are limited. OpenAI&rsquo;s <code>tiktoken</code> handles inference well, while HuggingFace&rsquo;s <code>tokenizers</code> supports training but has complexity and overhead. <strong>RustBPE</strong> is a Rust implementation that provides training capabilities with better performance.</p> <p><strong>RustBPE was developed by Andrej Karpathy</strong> as part of the <a href="https://github.com/karpathy/nanochat/tree/master/rustbpe" target="_blank" rel="noopener">nanochat project</a>. This analysis covers the RustBPE implementation, including its architecture, performance characteristics, and Python integration.</p>