Skip to content
Main Site News Console

99% KV Cache Reduction, 6x Single-GPU Throughput: HKUST, CMU, and Yanshan Tech Open-Source New Model

· 量子位
国内AI

Scan QR code to follow QbitAI

Image 1

< img id=“wx_img” src=“https://www.qbitai.com/wp-content/uploads/imgs/qbitai-logo-1.png” width=“400” height=“400”>

Image 2Yi Shui 2026-10-26 14:45:25 Source: QbitAI

A Single GPU Running 512 Concurrent Long-Inference Streams

Reported by Yunzhong

QbitAI | WeChat Official Account: QbitAI

A paper published by Google earlier this year managed to rattle the stock prices of memory chip giants.

The paper showed that the KV Cache can be compressed by at least 6x.

As large model inference becomes increasingly memory-hungry, the KV Cache represents a massive share of the overhead.

When Google demonstrated such a drastic reduction, shares of memory manufacturers like Micron tumbled.

Yet just a few months later, someone has taken an even more radical knife to the KV Cache.

This time, simple compression wasn’t enough.

Even as the context length stretches from 4K all the way to 256K, KV Cache memory consumption stays completely flat.

At a 256K context window, it remains strictly capped at 72 MiB. Compared to the base model, the KV Cache footprint is slashed by 99.1%.

Even more striking, a single 80GB GPU can now run 512 concurrent long-inference streams simultaneously, hitting an output throughput of 12,124 tokens/s—6.42 times that of the baseline model.

Crucially, reasoning performance remains largely intact: its average pass@1 rate on AIME 2024–2026 reached 82.9%, whereas the best competing bounded-state models peaked at just 18.8%, even at the 14B parameter scale.

Image 3

Meet EdgeRock-4B, freshly released and open-sourced by RockMatrix (a subsidiary of Yanshan Technology) in collaboration with institutions including HKUST and CMU.

The breakthrough it brings to long-context inference can be summed up in one sentence:

No matter how long the reasoning chain, memory never grows.

Model weights and inference code are now open source, with support for both text and multimodal image reasoning.

Why Does Longer Inference Cost More?

Modern models routinely spend tens of thousands of tokens “thinking”; AI agents can execute dozens of consecutive turns, continually packing search results, code snippets, and tool-call logs into the context.

The harder the model works, the sooner GPU memory hits a wall.

The culprit is the KV Cache.

Most modern architectures store the entire inference history inside the KV Cache, appending new key-value pairs with every single token generated.

Take Qwen3.5-4B as an example: each generated token adds 32 KiB to the KV Cache. By the time a sequence reaches 64K tokens, it consumes 2 GiB; at 256K tokens, that single sequence eats up 8 GiB.

Image 4

In the era of reasoning models, this is a fatal bottleneck.

Models are increasingly encouraged to engage in “deep thinking,” outputting tens of thousands of tokens per run. Meanwhile, agents loop through dozens of tool executions, blowing up the context window.

The longer the inference run, the more VRAM it consumes—and the fewer concurrent requests a single card can handle.

On edge devices, the KV Cache must also fight for scarce system memory against perception and control modules.

The industry has generally explored two paths to tackle this issue.

The first is slimming down each individual token:

Techniques like Multi-Head Latent Attention (MLA) compress KV projections, quantization methods like KIVI reduce precision, and Google’s TurboQuant follows a similar vein.

However, these approaches only reduce the per-token cost; total cache size still grows linearly with sequence length.

The second path involves bounded-state models:

Recurrent models, State Space Models (SSMs), and sliding-window attention naturally maintain a constant memory footprint.

The problem? They struggle to retain distant details critical for subsequent reasoning, severely handicapping their problem-solving capabilities.

This sets up the real challenge:

Can we retain critical reasoning details while keeping memory strictly bounded?

EdgeRock’s answer: Yes.

A Single GPU Running 512 Concurrent Long-Inference Streams

Let’s start with raw throughput.

Under a test benchmark of 32K input tokens and 32K output tokens, all models were evaluated on a single Hopper 80GB GPU, each tuned to its optimal empirical batch size.

EdgeRock-4B scaled smoothly up to a concurrency of 512, hitting an output throughput of 12,124 tokens/s.

In contrast, the fastest configuration for the baseline Qwen3.5-4B topped out at a batch size of 64, delivering only 1,889 tokens/s. Gemma 4 E4B, MiniCPM5-2B, and LFM2.5-2.6B all hovered between 1,700 and 2,200 tokens/s.

Image 5

Why such a massive performance gap? The math is straightforward.

After completing a 32K-input / 32K-output run, Qwen3.5-4B requires 2 GiB of KV Cache per sequence.

A single 80GB card saturates after only a few dozen concurrent streams, forcing subsequent requests into a queue.

EdgeRock-4B, on the other hand, caps the total historical state per sequence at a fixed 97.76 MiB during batch execution, completely invariant to length.

To put this into perspective: if Qwen3.5-4B were to run these 512 sequences concurrently, the KV Cache alone upon completion would demand 1 TiB—equivalent to the combined VRAM of roughly thirteen 80GB GPUs.

EdgeRock fits all of it onto a single card, peaking at 61.26 GiB VRAM—including model weights.

Image 6

Next, consider scalability.

Across 16 workload profiles pairing inputs and outputs from 4K to 32K, scaling the batch size from 32 to 512 boosted weighted decoding throughput from 4,128 to 12,497 tokens/s, while peak VRAM never breached the single-GPU ceiling.

Image 7

Single-stream latency is equally impressive.

Models where the KV Cache expands over time share a common flaw: generation slows down as the sequence grows.

Every new token requires attending across an ever-larger KV Cache.

With EdgeRock, computational cost per token does not scale with context length.

Following a 128K input context and generating another 128K tokens, its generation speed holds rock-solid at 233.7 tokens/s from start to finish.

Qwen3.5-4B, meanwhile, steadily drops from ~169 down to 151.3 tokens/s.

Image 8

The Longer the Context, the More You Save

Multiplying throughput is, from another angle, dividing cost.

On the same GPU and under identical workloads generating the same number of tokens:

  • At 4K input, EdgeRock requires only 24.9% of Qwen3.5-4B’s GPU time;
  • At 32K input, it requires a mere 15.6%.

Under standard GPU-hour billing, compute costs per token decrease proportionally.

Image 9

Averaged across four input lengths, EdgeRock delivers 4.94x the throughput of Qwen3.5-4B, and 7.18x that of LFM2.5-2.6B.

More important is the trajectory: the longer the context, the wider the gap becomes.

Qwen3.5-4B’s KV Cache swells indefinitely with sequence length, whereas EdgeRock’s stays completely constant.

Beyond raw savings, bounded memory brings another often-underappreciated benefit: predictability.

Exact memory requirements per request can be calculated before inference even begins.

  • Inference platforms can provision capacity strictly by request count, without guessing how long each session will run;
  • Edge hardware can allocate memory in a single upfront block, eliminating out-of-memory crashes during extended reasoning;
  • Agent memory overhead no longer scales with the number of tool invocations;
  • Dedicated ASICs can even store the entire historical state directly in on-chip SRAM.

As memory bandwidth and capacity remain at a premium, the economics of this approach will only become more compelling.

Slashed Memory, Zero Compromise on Intelligence

At this point, seasoned practitioners might ask: constant-memory architectures like RNNs and Mamba have been around for years—what makes this different?

What makes it different is that it cuts memory without crippling reasoning capacity.

The team benchmarked EdgeRock-4B against 16 bounded-state models spanning GDN, RWKV, and Mamba architectures, including three 14B-scale baselines.

On AIME 2024–2026, EdgeRock-4B achieved an average pass@1 accuracy of 82.9%. Among the control group, the top performer—RWKV-7 G1j 13.3B—managed only 18.8%, while AHN-GDN Qwen2.5 14B scored 11.4%.

Image 10

Across 29 language benchmarks, EdgeRock-4B took first place among bounded-state models on 24 tasks; its macro-average across all 29 benchmarks surpassed the strongest baseline, AHN-GDN Qwen2.5 14B, by 23.8 percentage points.

Compared against its unconstrained base model, capabilities remain largely preserved:

  • AIME pass@1 retains 96.3% of Qwen3.5-4B’s accuracy, with MATH-500 reaching 97.5%;
  • On 8 of the 29 language benchmarks—including MMLU-Pro, SuperGPQA, HumanEval, and MBPP—it actually surpassed Qwen3.5-4B;
  • Across six vision benchmarks, it averaged 84.9%, preserving 95.3% of Qwen3.5-4B’s multimodal capability.

These results place EdgeRock-4B on par with state-of-the-art small models whose memory scales unbounded with inference length.

The Secret: Dropping “Rock Anchors” into Memory

How does EdgeRock achieve this?

The core innovation is a novel memory mechanism dubbed Rock Memory Anchors (RMA).

Its design draws inspiration from rock climbing anchors: firmly pinning down vital information required for downstream reasoning.

Just as a climber drives anchors into a rock face to provide dependable footholds for every subsequent step, RMA “pins” essential details encountered throughout the reasoning trajectory into a fixed-capacity anchor bank, ready to be retrieved on demand.

Specifically, EdgeRock partitions sequence history into three dedicated components, each serving a distinct role:

  • Recurrent states summarize the evolving global context, capturing high-level intent;
  • A local attention window covers the most recent 1,024 tokens, managing immediate context;
  • The anchor bank preserves high-value details from deeper history, retaining critical specifics.

Image 11

The anchor selection pipeline works as follows:

A writer module generates candidate records for each token, which are scored by an evaluation network.

Every time 256 candidates accumulate, they are evaluated against existing entries in the anchor bank; only the top 1,024 scoring entries are retained, while the rest are pruned.

This process operates independently across four KV groups.

Retained anchors are mutually independent and static in content. Even after their original token positions slide out of the local attention window, subsequent inference steps can still attend directly to them.

In essence, the model does not attempt to remember everything; it learns to remember only what is worth keeping.

Combining the local window with the anchor bank fixes the attention KV footprint at a constant 72 MiB per sequence. This is the foundation of its flat memory consumption.

Architecturally, EdgeRock-4B is built on top of Qwen3.5-4B: it retains 24 Gated DeltaNet layers, replaces 8 full-attention layers with 1,024-token sliding window attention, and attaches dedicated RMA modules.

The entire model was adapted from Qwen3.5-4B via staged conversion, requiring zero pre-training from scratch.

Do these anchors actually make a functional difference?

The team conducted an ablation study: during inference, all 8 RMA modules were disabled, while recurrent layers and local attention continued to operate normally.

Across 256 long-context sequences spanning 16K to 130K tokens, four questions were posed per sequence covering retrieval, state tracking, cross-document reasoning, and variable binding:

  • With RMA enabled: the model answered 964 out of 1,024 questions correctly, achieving 94.1% accuracy;
  • With RMA disabled: it answered only 4 correctly, collapsing to 0.4% accuracy.

Image 12

Long-range recall virtually vanished without RMA, confirming that EdgeRock relies directly on these anchors to access distant context when generating answers.

Furthermore, because every anchor preserves its original position index, it is possible to inspect exactly what the model decided to remember.

Will Memory Demand Drop Further?

EdgeRock was developed collaboratively by RockMatrix (under Yanshan Technology), HKUST, CMU, and other institutions. The model is now open source and available for commercial use.

The team has shared that additional EdgeRock models across different parameter scales are on the way.

For long-horizon robotics, this architecture opens exciting possibilities.

A robotic system continuously sensing its surroundings and planning actions over hours or days cannot afford an endlessly expanding context buffer.

Predictable memory consumption during runtime removes a major hurdle for edge deployment.

Whether memory requirements will continue to tumble is a question for the market to answer.

One thing, however, is clear: models can now think for as long as needed without their memories blowing up.

Paper:
https://rockmatrix.ai/documents/edgerock-report-zh.pdf
GitHub:
https://github.com/rockmatrix-ai/EdgeRock
Hugging Face:
https://huggingface.co/rockmatrix-ai/EdgeRock

All rights reserved. Unauthorized reproduction or use in any form is strictly prohibited.

KV Cache Optimization

](https://www.qbitai.com/2026/10/505407.html#)

Scan to share via WeChat Moments

Google Open-Sources New RSI Breakthrough! Heavily Driven by Chinese Researchers 2026-10-19

Scan QR code to follow QbitAI Image 18

Tracking AI trends, reporting breakthrough moments in technology

Copyright © Beijing Geek Partner Technology Co., Ltd. All Rights Reserved. Beijing ICP Filing No. 17005886-1