Skip to content
Main Site News Console

Building a 4x R9700 Setup for a 10-Person Startup

· Reddit · r/LocalLLaMA Translated
Reddit

Quick share of a build I’m currently putting together for a client.

$18,000.

Specs:

Threadripper 9970x

128GB DDR5 ECC 5600

4x AMD Radeon R9700 AI Pro 32GB (128GB total VRAM)

1600W PSU (GPUs undervolted, <210W per card)

Inference engine: Running a fork of Radiance to serve Qwen 3.8 Next flash/27b and DSV4 Flash

Test Results:

16 concurrent sessions

Qwen 3.8 27b MXFP4

6.3k – 6.8k aggregate prefill tokens/s

900 tokens/s – 80 tokens/s aggregate decode (across 4k – 128k context lengths respectively)

Qwen 3.8 Next flash

48k/user | 6,880 tokens/s | 538.5 tokens/s aggregate prefill/decode

128k/user | 3,050 tokens/s | 365 tokens/s aggregate prefill/decode

All running with BF16 KV Cache.

Top Comments

MurmurRunner (⬆167): I feel like OpenAI and Anthropic are losing their grip. This is the 5th startup build I’ve seen running local models. ang3l12 (⬆83): Local models really only became both lightweight and capable enough to make financial sense for small businesses around this August. I used Qwen 3.8 27b back then to prove to management that we could run local models for most of our users. By the time the hardware arrived, glm 5.3 flash and Qwen 3.8 flash next validated our thesis even further, making our purchase of 4 Spark and 2 R9700 cards well worth it. CatalyticDragon (⬆37): Don’t sleep on the effort poured into inference evaluation harnesses and tooling. Without them, these models wouldn’t be anywhere near this competitive. mohelgamal (⬆23): People have been repeating this narrative for ages. But even if the models hit parity (which they haven’t yet), the big tech giants still have economies of scale and a massive base of low-frequency AI users—for whom paying $20/month makes way more sense than dropping $20k upfront on local hardware. It’s the exact same reason cloud computing took off before AI: companies could always buy and rack their own servers, but cloud delivered economies of scale and allowed spending to scale dynamically with actual demand. sleight42 (⬆57): Ha. At 16 users, it’ll take about 5.75 years to break even. Is data privacy the main concern here? sayamss (OP) (⬆9): It’s actually just 9 months. mohelgamal (⬆34): In two to three years, 99% of these startups will go the way of all startups (bust), and all this hardware will be sitting on eBay waiting for me to scoop it up for cheap. sayamss (OP) (⬆22): $18,000.