Skip to content
Main Site News Console

Qwen3.8-27B: R9700 Hits 159 tok/s, Strix Halo Reaches 64 tok/s

· Reddit · r/LocalLLaMA Translated
Reddit

LemonSeed Studio is an iPad editor/IDE that supports on-device inference using an AMD GPU connected via a Thunderbolt eGPU enclosure. It embeds unmodified upstream Linux amdgpu + amdkfd drivers (mac_linuxgpu) as a PCIDriverKit extension and runs the LemonSeed Engine (LSE) on the GPU. The pictured setup shows an iPad Pro + Sapphire Radeon AI PRO R9700 running Qwen3.8-27B Q4 with a Q8 DFlash2 draft model at a 131k context length.

LSE uses the exact same engine across all platforms (Linux, macOS, and iPadOS). It records the model’s forward pass as a compute graph, performs operator fusion, and generates kernels tailored to the host GPU. When multiple memory layouts exist for an operator kernel, it benchmarks them on-device and keeps the fastest one. Speculative decoding (supporting MTP and DFlash2) dynamically determines how many draft tokens to verify based on measured acceptance rates and overhead.

Decode speeds (code prompt, Qwen3.8-27B Q4, baseline / MTP=3 / DFlash2, in tok/s):

  • R9700 on iPadOS (LemonSeed Studio): 31.2 / 108.7 / 158.5

  • R9700 on macOS: 32.2 / 111.9 / 159.4

  • R9700 on Linux: 32.0 / 111.0 / 144.6

  • Strix Halo (Radeon 8060S) on Linux: 14.0 / 47.8 / 64.4

Prefill speeds at 4K tokens: 1,636 (iPadOS), 1,634 (macOS), 1,418 (Linux R9700), and 517 (Strix Halo). It holds up well at 32K tokens: 1,395 / 1,401 / 1,226 / 462.

What’s new in 0.5.8:

  • DFlash2 draft trees enabled by default: verifies an entire candidate tree in a single target model forward pass when empirically faster than chains
  • Strix Halo (gfx1151) prefill and decode optimizations: fused gate/up GEMM, 2 query tiles per workgroup in prefill attention
  • Fixed a GPU startup crash
  • Linux archives now bundle the HSA runtime, so only an amdgpu driver with access to /dev/kfd is required
  • lse-server models / pull: directly pulls Hugging Face models along with their accompanying MTP or DFlash2 draft models

Offers an OpenAI-compatible server, a CLI tool, and a C library (libLSE).

Engine: https://github.com/Geramy/LSE

Studio: https://github.com/Geramy/LemonSeed-Studio

Happy to answer any questions, especially regarding the details of driving an eGPU on iPadOS.

Top Discussions

harrro (⬆49): Today I learned an iPad can actually connect to a desktop-class GPU… Amazing project. TheOriginalG2 (OP) (⬆5): Thanks! t-i-m-o-t-h-y (⬆17): I have a question about getting an eGPU running on iPadOS… how on earth did you manage that?! TheOriginalG2 (OP) (⬆20): Haha, get an Apple developer account, request the DriverKit development entitlement (which is granted instantly), and then compile and locally deploy it to your provisioned iPad, haha. TheOriginalG2 (OP) (⬆5): Initially, I just ported amdgpu to Mac. Once that was done, I switched to wrapping Linux with a shim layer so I wouldn’t have to maintain amdgpu and its associated software. Thanks to a colleague who pointed out that this was a better approach, things became much simpler after that since I already had the framework from the original port. Now others can build on top of this shim layer to port more things to Mac. feelspeaceman (⬆4): Awesome, gufo also runs Q38-27-Q8 at 40 t/s with 600-700 t/s prefill. While not fully optimized yet, it’s already quite usable. After all, unless there’s a specific reason, Q38FN is the go-to model for Strix Halo anyway. Edit: Typo, meant Q38-27-Q4. TheOriginalG2 (OP) (⬆1): Haven’t tested it yet; I don’t really use gufo. TheOriginalG2 (OP) (⬆1): Yeah, it’s a fantastic model. Q4 and Q6 are usually my go-to choices, and this is still just the first round of optimization.