Skip to content
Main Site News Console

How to Cut Inference Costs by 77% in Production with Custom SLM Vocabularies

· Hugging Face Translated
教程模型卡

Published on October 21, 2026

Models specialized for specific domains or tasks by fine-tuning general-purpose open-source models like Qwen and Gemma inherit vocabularies designed for a broad array of domains and general use cases.

In an essay grader running in our production environment, this vocabulary split the text the model wrote most frequently into far too many tokens. This post previews select findings from our upcoming VocabX paper: how we specialized a vocabulary around the cost per token—rather than token frequency—and slashed measured inference costs by 77% without modifying the base model.

Figure 11: Same sentence, fewer tokens: 14 tokens with the base tokenizer, down to just 4 with VocabX

The tokenizer is typically treated as a fixed property of the base model, inherited and left untouched. Yet for a model dedicated to a single domain, it can be one of the highest-leverage cost drivers—locked in well before any training even begins.

One system we run in production is an automated Portuguese essay grader built on Qwen3-4B. It ingests an essay, a prompt, and reference materials, returning a JSON evaluation that includes scores and feedback across individual competency dimensions.

The model is already small (4 billion parameters / 4B) and purpose-trained for this task. However, it still uses the original Qwen3-4B tokenizer, which was built for dozens of languages and massive varieties of text. In the grader, it is used for just one language (Portuguese) and a single response schema. This imposes a twofold cost penalty: the grader burns unnecessarily more tokens generating text for each request, and for every token generated, the model must score a vocabulary of 151,669 entries—chock-full of tokens the task almost never touches.

We rebuilt this vocabulary using a method called VocabX, keeping the original base model, data, and training recipe intact. The numbers in this post are drawn from the version of the paper currently under review and may change before official publication.

A Vocabulary Built for Everyone

Mainstream language models read and write in tokens, and the tokenizer determines how text is broken down into those tokens. Tokenizers for general-purpose models are trained on massive corpora spanning multiple domains and languages—from source code and legal contracts to Chinese. For a general-purpose model, this makes sense.

However, for a model dedicated to a single domain, this creates two kinds of waste