Skip to content
Main Site News Console

First-Token Latency Cut by 32.4%: Shishi Tech Advances Cluster Scheduling to Boost Domestic Inference Efficiency

· 量子位
国内AI

METASTONE Unveils TokenFleet Scheduling Framework

In GPU clusters, request queuing and resource underutilization often occur simultaneously.

At the 2026 CCF National Annual Conference on High Performance Computing (CCF HPC China), METASTONE officially unveiled TokenFleet, a distributed scheduling framework. TokenFleet focuses its optimizations on request routing, cross-datacenter capacity coordination, and mixed-workload governance—aiming to maximize the effective yield of existing compute resources.

If an LLM inference service is likened to a “Token optimization factory,” GPUs are the manufacturing equipment, inference instances are the workstations, and incoming business requests are production work orders. TokenFleet’s core mission is to match these orders with available production capacity with high precision and efficiency.

Given the exact same hardware infrastructure, different workload characteristics and allocation strategies can lead to vastly different queue times and SLA-compliant output. Delivering compute services at scale demands a holistic approach to production scheduling.

TokenFleet: Focusing on Cluster Effective Output

Once models are deployed into production, clusters must typically handle a diverse array of business scenarios concurrently.

When multiple models share cluster compute resources, different tenants bring distinct Service Level Agreements (SLAs): real-time chat demands millisecond-level low latency, long-document analysis heavily consumes compute and VRAM, while offline batch jobs are governed by strict deadlines. Furthermore, when serving nodes are distributed across multiple datacenters, orchestrating capacity seamlessly across regions becomes an additional challenge.

Crude traffic distribution strategies often cause long queues on some instances while leaving other nodes underutilized. Resource contention between long-context requests and batch tasks degrades interactive response times. Worse, request timeouts, cancellations, and redundant recomputations squander precious GPU cycles without producing meaningful deliverables.

While maintaining consistent model quality, service requirements, and compute budgets, TokenFleet aims to increase effective token output and reduce the overall cost per effective unit of output.

It is driven by three collaborative core technologies:

  • TokenPilot: Selects optimal inference instances within the same model pool;
  • FleetMind: Orchestrates macro-level capacity coordination across datacenters;
  • SLOMax: Governs mixed workloads through admission control, queue management, and capacity allocation.

How requests are routed, when tasks enter queues, and how cross-datacenter capacity is mobilized all converge on a single key metric: effective token output that is successfully delivered in compliance with SLAs.

Image 3
△ Figure 1: Overview of TokenFleet’s three core scheduling technologies

TokenPilot: Balancing Cache Gains Against Queuing Costs

During request routing and instance selection, which instance should handle an incoming request?

A brief Q&A and a lengthy document both count as a single request statistically, yet their compute and VRAM footprints differ by orders of magnitude. Naive load balancing based solely on request count fails to evenly distribute the true workload across instances.

The KV cache mechanism introduces even more nuanced trade-offs: routing requests with shared long prefixes to an instance holding the cached KV states allows reuse of past computations, drastically cutting redundant calculations during the prefill phase (processing input context).

However, instances with high cache hit rates often suffer from greater concurrent queuing risks. If traffic skews excessively toward a single instance, the computation time saved can easily be cancelled out by prolonged queue delays.

When selecting instances, TokenPilot weighs the benefits of cache reuse directly against queuing costs. The system estimates computational workload using input context length, output budgets, and reusable prefixes. It then scores and filters candidate instances by holistically evaluating queue depth, in-flight tokens, KV cache memory pressure, and predicted latency. (Note: Output budget serves solely as a routing constraint and does not necessarily equal the actual final output length.)

With these metrics, the scheduler quantifies and compares the expected processing costs across instances, accurately determining whether the compute savings from cache reuse outweigh any additional queuing delays.

Image 4
△ Figure 2: TokenPilot intelligent inference routing framework

According to GLM-5.3 benchmark tuning data released by METASTONE, under tested hardware clusters and specific workload conditions, switching to predicted-latency-based routing resulted in a 32.4% reduction in P95 Time to First Token (TTFT) and a 71.3% reduction in P50 TTFT.

Image 5
△ Table 1: TokenPilot Time to First Token (TTFT) tuning results. Source: METASTONE conference presentation materials. Improvement rate calculated as (Baseline - Optimized) / Baseline; latency reductions do not equate directly to cost reductions.

Here, P50 reflects median scheduling response times, while P95 captures tail latency. While these metrics showcase response time improvements in a benchmark environment, comprehensive cost-benefit assessments still require factoring in overall system throughput and actual resource utilization.

FleetMind: Calculating Real-World Available Capacity Across Datacenters

When the scheduling scope expands from a single cluster to multiple datacenters, the decision layer must incorporate global capacity awareness.

An idle GPU on a node does not necessarily mean it can serve incoming traffic immediately. Factors such as whether the target model is loaded, weight versions match, serving endpoints are healthy, and cross-region network latency all heavily impact final delivery performance.

At its core, FleetMind evaluates genuinely usable model serving capacity. Within the TokenFleet architecture, it aggregates model deployment statuses, real-time capacity headroom, and network links across datacenters. It first determines the destination datacenter at a macro level, then delegates the request to that datacenter’s local scheduling engine to assign a specific inference instance.

When queues build up in a hotspot datacenter, the system dynamically assesses whether remote sites host ready endpoints running identical model versions with surplus compute, determining whether compliant new requests can be dispatched cross-region.

Whether a request can overflow to another region is governed first by data compliance policies and geographic access restrictions. Whether it is worth overflowing depends on a trade-off between backhaul latency, queuing delay, compute load, cache affinity, and cross-region bandwidth overhead.

Cross-datacenter coordination operates across multiple time horizons: short-term fine-grained routing adjustments for new requests, alongside medium-to-long-term orchestration of model replica distribution and node pre-warming. Combined with capacity reservation, load-cap awareness, and automated failover fallbacks, this minimizes capacity mismatches.

Image 6
△ Figure 3: FleetMind cross-datacenter execution framework. Note: Region A serves as the traffic ingress point and is not part of the GPU capacity pool; Region B represents existing clusters, while Regions C and D are expansion datacenters.

While TokenPilot and SLOMax emphasize granular governance within existing clusters, FleetMind empowers horizontal elastic scaling across newly added compute sites. The primary objective of cross-datacenter scheduling is to unlock fragmented, idle compute to absorb sudden traffic spikes under strict compliance constraints, eliminating data-silo inefficiencies and redundant capacity reservations.

SLOMax: Precision Governance of Mixed Workloads by Service Level

Workloads arriving at the same cluster often carry vastly different SLA requirements.

Interactive chat demands ultra-low TTFT and smooth token generation rates; long-context tasks depend on heavy prefill throughput and ample KV cache memory; and offline batch processing prioritizes deadlines and overall batch throughput. Mixing these heterogeneous workloads in a single queue inevitably triggers fierce resource contention.

Extended prefill durations, KV cache bloat, and highly concurrent batch jobs can crowd out interactive requests. The resulting timeouts, client cancellations, and retries waste valuable compute cycles on work that never reaches completion.

SLOMax analyzes input sizes, output budgets, task types, and tenant Service Level Objectives (SLOs) to profile true workload characteristics. It then applies differentiated admission control, pooled queuing, and dynamic capacity rebalancing. By prioritizing capacity for latency-sensitive interactions, it effectively isolates the performance jitter caused by co-locating short and long requests.

Image 7
△ Figure 4: SLOMax classified queuing and capacity feedback framework. Note: Performance feedback dynamically adjusts capacity quotas and admission thresholds; the framework does not assume hard preemption of in-flight tasks.

When evaluating overall scheduling efficiency, TokenFleet establishes “compliant yield” as its key optimization metric.

Goodput (compliant output throughput) is defined as: the total volume of valid tokens generated by requests that are successfully delivered and meet all designated service level requirements over a complete test period, divided by the test duration. Separate compliance criteria and statistics are maintained for distinct workload types.

The system feeds real-time metrics—including compliant throughput, failure rates, cancellation rates, and recomputation overhead—back to the scheduling core to dynamically refine admission thresholds and compute allocations. Evaluating cluster efficiency requires looking beyond raw GPU utilization and peak throughput to focus on the actual volume of successfully fulfilled, SLA-compliant work.

In METASTONE’s published GLM-5.3 estimation model, introducing collaborative flow control scheduling led to a modest ~1.5% increase in total cluster output throughput, while compliant output throughput (Goodput) surged by approximately 29.4%, raising the overall SLA compliance rate from roughly 67% to around 86%.

Image 8
△ Table 2: SLOMax effective token throughput gain estimation example. Source: METASTONE conference presentation materials. This table represents an illustrative projection; compliance rate uses original rounded values, reflecting an increase of approximately 19 percentage points.

These figures serve as an illustrative model demonstrating the scheduler’s potential in boosting delivery quality; actual commercial gains must be verified against specific production workloads and service metrics.

Given comparable total output volumes, drastically reducing timeout drops and redundant retries frees up substantial capacity for compliant token delivery. SLOMax leverages this closed-loop metrics feedback to continuously refine subsequent global resource allocations.

The Efficiency of the Token Optimization Factory Hinges on Scheduling

TokenFleet advances LLM inference performance optimization to the distributed cluster level. Effectively allocating single-instance performance across multiple requests and seamlessly uniting compute power across geographically dispersed datacenters requires an intelligent scheduling framework that tightly binds underlying hardware infrastructure to high-level business SLAs.

METASTONE positions itself as an independent, third-party full-stack compute operator. Not tied to any single model ecosystem, it specializes in delivering end-to-end compute solutions for enterprise clients—spanning hardware selection, network architecture design, task scheduling, performance optimization, and cluster operations.

Drawing on deep expertise rooted in national supercomputing centers, METASTONE maintains an operational SLA exceeding 99.95% across its compute clusters. Today, METASTONE manages over 20,000 PFLOPS of heterogeneous compute resources and operates more than 10 supercomputing and AI computing centers (including two national supercomputing centers). Its core team members have won the prestigious Gordon Bell Prize—often dubbed the “Nobel Prize of Supercomputing”—three times, bringing extensive real-world experience to full-stack delivery of ultra-large-scale heterogeneous compute clusters.

Centered on the concept of a “Token optimization factory,” METASTONE refines its foundational system software through an in-house development and deployment closed loop. This effort aims to drive down the marginal cost of high-frequency, general-purpose LLM inference while unleashing the full practical potential of domestic hardware.

While its previously released Meta-Infer engine focused on squeezing maximum computational efficiency from individual inference instances, the newly unveiled TokenFleet shifts the focus to distributed cluster scheduling. By delivering precise request routing, elastic cross-region capacity coordination, and fine-grained mixed-workload governance, it provides the technological bedrock for large-scale production.

Empowering more enterprises to adopt and maximize LLM capabilities affordably and effectively requires both advanced hardware for raw compute and robust system software to organize vast cluster capacity efficiently. TokenFleet’s exploration translates the mission of cost reduction and efficiency gains into reality—grounded in the optimal routing of every single request and the maximal release of every ounce of compute capacity.