Skip to content
Main Site News Console

Baidu Open-Sources Wenxin 5.5 Flash Base, Setting New SOTA for Its Size Across Multiple Benchmarks

· 量子位
国内AI

Baidu Officially Open-Sources Latest Foundation Model ERNIE 5.5 Flash Base

On November 5, Baidu officially open-sourced its latest foundation model, ERNIE 5.5 Flash Base. Featuring a Mixture-of-Experts (MoE) architecture, the model boasts a total of 280 billion parameters, with 14 billion activated per token. Pre-trained on 33.5 trillion tokens, it supports a pre-training context window of up to 128K tokens and can scale to an ultra-long context of up to 1 million tokens during fine-tuning and inference. Across multiple benchmarks evaluating knowledge, reasoning, mathematics, coding, multilingual capabilities, and long-context performance, ERNIE 5.5 Flash Base achieved state-of-the-art (SOTA) results among models of comparable scale, outperforming peers such as DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base.

Open-source repository: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base

Image 3

Image 4

As large language model applications expand beyond simple text Q&A to long-document analysis, complex multi-turn interactions, and agentic workflows, the volume of information models must process continues to surge—driving up computational and memory overhead substantially. Consequently, drastically boosting resource utilization efficiency while maintaining core model capabilities has emerged as a central challenge in LLM research and development.

Focusing sharply on efficiency, ERNIE 5.5 Flash Base introduces systematic optimizations and algorithmic innovations across key technical components, including the attention mechanism, expert routing, and training paradigm:

  • Attention Mechanism: The ERNIE team introduced Latent-Gated Sparse Attention, which deeply integrates Hybrid Sparse Attention, Low-Rank Gated Attention, and Virtual Head Attention.

    • By synergistically combining HCA with MLA, distinct attention layers are assigned dedicated roles: compressing historical context versus enabling fine-grained retrieval of past positions;
    • Low-Rank Gated Attention computes information gates using existing low-dimensional representations, slashing the gating projection parameter count by 75%;
    • Virtual Head Attention (VHA) enhances expressiveness and training quality while seamlessly fusing into attention kernels during inference with virtually zero additional overhead. Together, these designs dramatically alleviate the resource burden of long-context processing while fully preserving the model’s ability to capture and process critical information.
  • Expert Routing Architecture: The model adopts a Multi-Router Latent MoE architecture, deploying multiple independent routers to collaboratively select experts. This markedly improves the correlation between routing scores and actual expert utility. Compared to conventional single-router approaches, this architecture mitigates single-point scoring bias without increasing the number of activated experts, computation per expert, or communication overhead.

  • Training Objectives & Inference Decoding: The model introduces the ECHO mechanism. Instead of mounting a separate, auxiliary MTP (Multi-Token Prediction) module, it directly reuses the final layer of the backbone network. This encourages the backbone to autonomously learn to predict future tokens and plan sequence trajectories in advance, significantly boosting logical reasoning and code generation capabilities. Trained with just single-step ECHO during pre-training, ERNIE 5.5 Flash can leverage this reused final layer as a high-performance draft model for speculative decoding. It demonstrates remarkable multi-step extrapolation capability—delivering drafting quality far superior to traditional auxiliary MTP modules trained with the same single-step setup.

Beyond architectural innovations, co-design with underlying infrastructure played an equally vital role. The ERNIE team deeply optimized a distributed Muon optimizer and introduced FlashMask-4, an attention operator custom-tailored for large head dimensions. Together, they significantly reduced communication overhead, improved VRAM efficiency, and boosted overall computational throughput during ultra-large-scale training. This underscores that breakthroughs in model capability hinge not only on algorithmic design, but also on full-stack synergy with high-efficiency training systems.

Foundation models not only encapsulate broad cognitive capabilities acquired during pre-training, but also serve as the bedrock for downstream fine-tuning and application innovation. Unlike end-user application models, base models grant developers and enterprises full flexibility to customize and fine-tune solutions tailored to specific business domains and vertical applications. Baidu’s open-source release provides global developers with a more competitive technical foundation while substantially lowering the training costs and barriers to entry for secondary model development.

Today, competition in the AI industry has entered a new phase characterized by the simultaneous advancement of capability, training efficiency, and real-world deployment. As reasoning models and agentic applications rapidly evolve, industry demand for foundation models has shifted from simply pushing benchmark boundaries toward more efficient training paradigms, flexible capacity expansion, and adaptive domain transfer. While achieving leaps in core capability and breakthroughs in training/inference efficiency, ERNIE 5.5 Flash Base delivers an extensible technical foundation for developers and the open-source community—accelerating diverse explorations and practical deployments across the broader LLM ecosystem.