Choosing a retry budget for an OpenAI-compatible client

Retries are useful only when they are bounded. A client that retries every failure can turn a short outage into a request storm, especially when several coding agents share one endpoint.

Separate transport failures from application failures

Retry connection resets, DNS timeouts, and HTTP 408/429/500/502/503/504 responses. Do not automatically retry malformed requests, authentication failures, or permission errors. Those need a configuration or credential fix.

Use a small exponential schedule

A workable default is five attempts with delays of 1, 2, 4, 8, and 16 seconds, plus random jitter of 0–25 percent. Cap the total retry time below the caller’s interactive timeout. For a batch worker, a longer deadline is reasonable; for an editor completion, it usually is not.

Track the attempt count and the provider response ID in logs. This makes it possible to distinguish a server-side rate limit from a client that duplicated a request. For non-idempotent operations, attach an idempotency key or avoid retrying after the server may have accepted the request.

Make the budget observable

Record these fields per request: model, HTTP status, latency, attempt number, total elapsed time, and whether the response was served after a retry. A simple alert is the percentage of successful responses that needed a retry. If that rises, lowering concurrency is often safer than raising the retry limit.

When testing a second OpenAI-compatible endpoint, keep the request body and client timeout constant. APIClaw’s developer documentation describes its OpenAI-compatible base URL and client setup. That lets you compare transport behavior without silently changing the application layer.

Checklist

  1. Classify status codes before retrying.
  2. Apply exponential backoff with jitter.
  3. Enforce a total deadline.
  4. Use idempotency protection for writes.
  5. Log attempts and response identifiers.
  6. Reduce concurrency when rate limits persist.

The goal is not the highest retry count. It is a bounded recovery path that fails clearly when the upstream cannot answer.