Skip to content
Main Site News Console

[AINews] DeepSeek Open-Sources SOTA Inference Framework DeepInfer: 155T Daily Tokens per Cluster, '2M Daily Revenue, ~80% Margin, 81-Day Payback

· Latent Space Translated
播客深度访谈

Back in April, we covered The Inference Inflection. Historically, most of the breakthroughs have come from closed-source decacorns spanning from Fireworks to Baseten to Modal.

Now, open source has answered back. Right on day one of the PyTorch Conference, DeepSeek fired an ICBM missile into the discourse—officially open-sourcing its production serving stack, DeepInfer. It supports prompt caching for up to 72 hours, compared to the 12-hour window OpenAI recently unveiled:

🚀 DeepSeek (@deepseek_ai): Introducing DeepInfer, our in-house inference framework is now fully open-sourced!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We are sharing our production-grade stack to help the community cut compute costs, reduce energy consumption, and make AI more…

It is their customization based on vLLM, with acknowledgments to SGLang and TRT-LLM, unlocking a massive set of new features… though without their “secret sauce”—namely, ultra-optimized KV caching (compact storage + >8-hour TTL), Prefill-Decode (PD) Disaggregation, and load balancing (their 3FS was open-sourced recently). We translated their official blog post and published it here, complete with a research guide, explainers, a codemap, and a subsystem guide: