Service Availability
High Availability Architecture
- Multiple upstream redundancy: The same model is connected to multiple upstreams/multi-region compute resources, with automatic failover when a single point fluctuates;
- Health checks: Continuously probe the latency and success rate of each upstream; abnormal channels are automatically removed and automatically restored after recovery;
- Automatic retries: Instant upstream errors are retried first within the gateway, minimizing the exposure of jitter to the caller.
Behavior During Outages
- When a specific model’s upstream experiences a widespread failure, that model may see increased latency or more 5xx errors, while other models remain unaffected — we recommend configuring fallback models for critical business workloads;
- Planned gateway maintenance will be announced in advance in the console’s announcement section.
Recommended Client Configuration
| Item | Recommended Value |
|---|---|
| Connection timeout | 5–10s |
| Read timeout (non-streaming) | 60–120s |
| Read timeout (streaming/inference models) | ≥300s |
| Retries | 429/5xx exponential backoff 3–5 times |
If you encounter persistent issues, please contact support with the time range of the error and error.message for troubleshooting.