Rate Limiting and 429

Three Reasons for 429

1. Request rate too high: concurrent requests/QPS exceed the current account tier within a short period;
2. Insufficient token quota: the quota limit set for the token has been used up;
3. Upstream model rate limiting: when certain model upstreams are congested and the gateway has already retried automatically but is still unavailable, the error is passed downstream.

The response body’s error.message will explain the specific reason.

The Correct Retry Approach

1st failure → wait 1s + random jitter → retry
2nd failure → wait 2s + random jitter → retry
3rd failure → wait 4s + random jitter → retry; if it still fails, alert

• Be sure to add random jitter to avoid a thundering herd when multiple clients retry on the same beat;
• For batch jobs, use a queue to control concurrency (for example, start with 5–10 concurrent requests and adjust after observing the success rate);
• Retrying quota-related 429s is ineffective; you need to top up your account or increase the token limit first.

Reducing the Likelihood of Triggering It

• Shift large batch jobs to off-peak hours;
• Combine long-text tasks to reduce the number of requests (do a bit more in one request instead of sending frequent small requests);
• Configure fallback models for degradable scenarios so the system automatically switches when the primary model returns 429;
• Enterprise users who need a higher concurrency tier should contact customer support through the console to request a quota increase.