Presented by SerpAPI: Today’s drop is packed with information, but you can scrape it all and convert it into Markdown format tailored for your agents! Save 50% on token costs compared to JSON.
We last covered the inference inflection back in April, where historically most of the breakthroughs came from closed-source decacorns, ranging from Fireworks to Baseten to Modal.
Now, open source is striking back. Today marks Day 1 of the PyTorch Conference, and DeepSeek seized the moment to drop an intercontinental ballistic missile, electrifying the floor: they have open-sourced their production-grade stack, DeepInfer. It supports prompt caching for up to 72 hours—by contrast, OpenAI recently announced their retention period at just 12 hours:
This is their heavily customized fork of vLLM (with nods to SGLang and TRT-LLM), packed with a massive wave of new features… yet stripping out their primary crown jewels—the hyper-optimized KV cache (compact format + 8+ hour lifespan), Prefill/Decode Disaggregation (PD separation), and load balancing (their distributed storage system 3FS was open-sourced earlier). We translated their official post and published it here, along with a comprehensive Research Guide, deep dive, Codemap, and subsystem guide:

As we noted in our breakdown of their supplementary disclosures, they claim to process 155T (trillion) tokens per day, generating $38/hour per SM100/A950 GPU even at off-peak rates (pricing is currently $0.15 / $0.60 per million input / output tokens: 155M million tokens * $0.14 blended rate / 24 hours yields ~$1M/hour in revenue). According to their published production figures (which we translated): each Prefill GPU processes 1.5M cached input tokens per second (with a 96% prompt cache hit rate), while each Decode GPU outputs 14k tokens per second across a cluster of ~2,400 GPUs:
