Skip to content
Main Site News Console

GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference

· Reddit · r/LocalLLaMA Translated
Reddit

Blog Post: ML Drift: Next-Generation GPU AI/ML Inference on the Edge - Google Developers Blog

The Google AI Edge team is thrilled to announce the open-source release of ML Drift under the Apache 2.0 license. It is our high-performance, cross-platform on-device GPU compute engine built specifically for client-side AI/ML inference. By abstracting the hardware and low-level API complexities across heterogeneous client GPUs—such as OpenGL ES, OpenCL, Metal, and WebGPU—ML Drift empowers developers to build real-time, interactive machine learning experiences across multiple platforms, ranging from advanced video effects to generative AI. ML Drift not only serves as the core built-in GPU acceleration engine in LiteRT, but can also be used as a standalone library for custom graphics and inference runtimes, providing a unified foundation to unlock peak performance across platforms.

Top Discussions

Chromix_ (⬆46): The big news isn’t “GPU acceleration supported” (that’s been around for years). What really catches the eye is the claim: “achieving an order-of-magnitude performance improvement compared to existing open-source GPU inference engines.” However, the benchmarks shown in the blog post are mostly on mobile platforms. The numbers on Intel’s latest mobile CPUs/GPUs do look decent, but there are no comparisons against CUDA- and Vulkan-based engines like vLLM or llama.cpp.

pmttyji (OP) (⬆42): Wondering what specific design gives their engine that extra prefill boost. We could really use that to improve LLM performance for llama.cpp on desktop. Prefill 8192, decode 1024, context length 9216, 4-bit weights (block size 32), fp16 KV cache, argmax sampling, with a sync after every generated token.

pseudonerv (⬆12): It also depends on the quantization scheme—certain quantization formats have extremely low compute overhead.

spaceman_ (⬆12): Interesting that they picked OpenCL as the primary target for Android. OpenCL used to have broad support on Android, but it has taken a back seat in recent years. I would have thought they’d lean more toward Vulkan compute shaders at this point.

IngwiePhoenix (⬆8): Only Metal, GL/GLES, and Vulkan… An intriguing choice, but they probably have their reasons. That unified kernel language looks promising, though. The only question is… why? What pain point in existing tools wasn’t being addressed that made them roll their own? o.O

Open-Adhesiveness-86 (⬆6): Picking OpenCL is largely driven by driver realities. Adreno’s OpenCL implementation allows keeping weights resident in texture memory and running fp16 math with fp32 accumulation at very low overhead; meanwhile, Mali’s support for Vulkan subgroups/coopmat across hardware generations is still a total lottery. The downside is that libOpenCL.so was never a standard NDK library—you have to dlopen it, and on some devices, it doesn’t even exist, which is why a fallback to GLES is necessary.

GradatimRecovery (⬆6): LiteRT is genuinely incredible. The e4b/e2b models on the edge gallery are easily the best on iOS.

pmttyji (OP) (⬆8): Not really. It’s better suited for edge devices.