Today, we are releasing two new encoder models on Hugging Face: LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. They deliver quality comparable to much larger models while maintaining high speeds as inputs grow longer. This means you can run document-level tasks on existing hardware, even on a CPU alone.
What you get:
- Small yet powerful: Matches or outperforms larger encoders on GLUE, SuperGLUE, and multilingual tasks.
- 8,192-token context length, with latency that scales very slowly as inputs grow.
- Fast on CPU: Approximately 3.7x faster than ModernBERT-base under long context lengths.
With these models, you can build intent routers, policy checkers, PII detectors, and text classifiers that run 24/7 at low cost. Check out the live demos below.
Why We Built General-Purpose Encoders
Last month, we released LFM2.5-Retrievers for multilingual search. LFM2.5-Encoders come from the same family but are much more versatile. They are pre-trained using a masked language objective, allowing you to fine-tune them for classification, token-level tasks, and search. Search is just one facet of an encoder’s capabilities, which is why we built general-purpose models instead of repurposing retrievers.
Encoders power many modern production NLP applications: classifiers, intent routers, and safety filters. These tasks typically run 24/7, are often deployed on CPUs, and handle increasingly long inputs. BERT laid the foundation for these models, and the recent ModernBERT has pushed the boundaries of accuracy, speed, and context length. LFM2.5-Encoders take the next step based on the LFM2 architecture: costs scale only slowly as inputs grow.
How the Encoders Were Built
We initialized the encoders from their respective LFM2 decoder backbones: LFM2.5-230M and LFM2.5-350M. We then converted each causal decoder into a bidirectional encoder with a few modifications:
- Bidirectional attention mask: Each token can now attend to tokens on both its left and right, rather than just preceding tokens.
- Non-causal short convolutions: We use symmetric padding so that each token’s convolution blends information from both its left and right neighbors.
- Masked language modeling: During training, we mask 30% of the tokens.
We trained both models in two stages:
- General language capability: Short-context masked language modeling pre-training at a 1,024-token context length on a large web corpus.
- Long-context adaptation: Extending the context window to 8,192 tokens and training on a full data mix to enhance factual, legal, and multilingual capabilities.
Evaluation Results
We fully fine-tuned the models on each task and reported the final scores. The complete benchmark includes 14 models across 17 dataset tasks from GLUE, SuperGLUE, and multilingual classification.
We report the average across 5 random seeds, ensuring stable results across different runs. The complete evaluation framework and raw results have been open-sourced.
LFM2.5-Encoder-350M ranks fourth among the 14 models. The three models ahead of it are all larger, including a 3.5-billion parameter model that is nearly 10 times its size. LFM2.5-Encoder-230M outperforms ModernBERT-base and all EuroBERT models, despite being smaller than most of them. Both models also score significantly higher here than our own LFM2.5-Retrievers.
Inference Speed on CPU and GPU
Our encoders inherit the high-speed inference capabilities of the LFM2 backbone. Since both our encoders and ModernBERT support an 8,192-token context length, we measured speed across the entire range.
Our encoders show the most significant advantage on CPUs. Here, LFM2.5-Encoder-230M is the fastest across all sequence lengths (even outperforming the smaller ModernBERT-base on short inputs). As input length increases, ModernBERT’s throughput drops sharply, whereas our LFM2.5-Encoders rise initially before leveling off. At 8,192 tokens, ModernBERT-base takes over a minute and a half per forward pass, while LFM2.5-Encoder-230M takes only about 28 seconds. This is roughly 3.7x faster. For developers, this means you can scan or classify an entire contract, a full chat transcript, or a long customer support ticket thread in under 30 seconds on a laptop CPU.
A similar trend is observed on GPUs, though with a smaller margin: on Apple GPUs, ModernBERT-base leads below roughly 1K tokens. Our encoders take the lead starting around 2K tokens. This shows that for long inputs, LFM2.5-Encoders are the faster choice, and the advantage becomes even more pronounced if you run on CPUs.
LFM2.5-Encoder Demos
The following demos are built on fine-tuned LFM2