How to handle OpenAI API 429 Too Many Requests

429 Too Many Requests It doesn't just mean "requested too quickly". In the LLM API, it may also come from excessive concurrency, token rate, account quota, busy single model, or double retries between the application and the SDK.

Save complete error evidence first

Record at least the following fields:

Occurrence time：
model ID：
HTTP Status：429
Error type / code：
Retry-After：
Request ID：
Number of concurrencies at that time：
input Token Estimate：
Number of retries：

If the balance, quota or account status clearly appears in the error text, deal with the account problem first; if rate limit, requests per minute or tokens per minute appear, adjust the request rhythm.

Five common sources of 429

Start by sending a short, non-streaming request using the same model. If a single request also persists for 429, check the quota and model status first; if it fails only during concurrency, check the queue and rate limit first.

Correct retreat strategy

Server returns Retry-After Priority is given to waiting by this value. This request header may be either a number of seconds or an HTTP date; it should first be parsed into the number of seconds to wait. Without this field, exponential backoff with random jitter is used:

No. 1 times：approx. 1 seconds
No. 2 times：approx. 2 seconds
No. 3 times：approx. 4 seconds
Stops after reaching the maximum number of attempts and returns an explicit error

Don't have all instances retry at the same second. Random jitter can reduce the second traffic spike caused by synchronous retries.

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
import random
import time

MAX_WAIT_SECONDS = 60.0

def parse_retry_after(value: str | None) -> float | None:
    if not value:
        return None
    try:
        return max(0.0, float(value))
    except ValueError:
        try:
            retry_at = parsedate_to_datetime(value)
            if retry_at.tzinfo is None:
                retry_at = retry_at.replace(tzinfo=timezone.utc)
            return max(0.0, (retry_at - datetime.now(timezone.utc)).total_seconds())
        except (TypeError, ValueError, OverflowError):
            return None

def backoff(attempt: int, retry_after_header: str | None = None) -> None:
    retry_after = parse_retry_after(retry_after_header)
    if retry_after is not None and retry_after > MAX_WAIT_SECONDS:
        raise TimeoutError("Retry-After Exceeds the allowed waiting time for the current task")
    if retry_after is not None:
        delay = retry_after
    else:
        delay = min(8.0, 2 ** attempt) + random.uniform(0, 0.5)
    time.sleep(delay)

This code is only responsible for parsing waiting times and pauses;attempt from 0 Start. Whether to retry depends on the error type, request idempotence, business deadline and maximum number of attempts.

Avoid repeated retries between the SDK and business layer

By default, OpenAI Python SDK will automatically retry 2 times after the initial request, that is, each SDK call will be tried up to 3 times. If the business layer calls the SDK 4 times in total, it may issue 12 upstream requests at worst.

When building your own retryer, you should explicitly keep only one layer:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://www.aifast.club/v1",
    api_key=os.environ["AIFAST_API_KEY"],
    max_retries=0,
)

If you use the SDK default retry, the business layer should not unconditionally loop on the same exception.

How to limit concurrency between batch processing and Agent

Batch processing

• Put the task into the queue;
• Set a fixed concurrency upper limit;
• Move up one gear at a time and watch 429;
• Record the number of successes, the number of 429s and Tokens, instead of just looking at the number of tasks.

Agent and tool invocation

A user task may trigger multiple rounds of model requests and tool postbacks. Limiting "user concurrency" is not the same as limiting "model request concurrency". Should also be set:

• The most LLM round for a single task;
• Maximum number of tool calls for a single task;
• Global model request concurrency;
• The maximum number of retries after failure;
• Single task budget limit.

When not to retry

• The error text states that the balance is insufficient or the account is unavailable;
• The request has reached the business deadline;
• It is impossible to confirm whether operations with side effects have been executed;
• The same model continues to fail and the maximum number of attempts has been reached;
• The business layer does not have idempotent keys, and retrying may result in repeated tasks or repeated deductions.

Acceptance criteria for repair completion

1. Single request and target concurrency complete at least one round of verification respectively;
2. 429 will read Retry-After or enter a capped retreat;
3. There is only a clear set of retry responsibilities between the SDK and the business layer;
4. Monitoring can distinguish the number of requests, tokens, concurrency and retries;
5. It can fail quickly after reaching the retry limit and will not wait indefinitely.

If short requests also persist 429, check firstCurrent models and pricesAnd check the account status; if the error becomes 502, disconnected or read timeout, continue to check502 and stream disconnected troubleshooting。