Skip to content

Rate Limit Handling: Building HTTP Clients That Survive 429s

Rate Limit Handling: Building HTTP Clients That Survive 429s

Background and Motivation

Every non-trivial upstream API imposes a rate limit. Sooner or later your client will receive an HTTP 429 Too Many Requests response (defined by RFC 6585) and what you do next separates a fragile script from a reliable service. The naive instinct — retry immediately, and faster if it fails again — is precisely the behavior that turns a brief throttle into a multi-minute outage. The AWS Builders’ Library article on retries and backoff (Brooker, AWS Builder Center) sums up the rule succinctly: “retries are selfish.” Each retry spends the server’s resources to increase one client’s chance of success. A few well-placed retries absorb transient failure; thousands of coordinated retries amplify it.

The motivation for a deliberate rate-limit handling strategy is therefore twofold. First, protect the upstream: avoid being the client that triggers the “thundering herd” pattern, where many callers back off to the same wall-clock instant and re-trigger the limit. Second, protect ourselves: surface a predictable error to the application layer instead of silently burning token budgets, hanging connections, or worsening latency. This article presents a layered approach — backoff → bucketing → proxy — that you can apply incrementally as your traffic grows.

Prerequisites

Before implementing, align on a few foundations:

  • A working HTTP client (we use Python + httpx for examples, but the patterns transfer directly to Go, Rust, TypeScript, etc.).
  • Ability to read and honor the Retry-After response header — the single most overlooked signal returned by a well-behaved server (Retry-After: 30 for seconds, or Retry-After: Thu, 01 Jan 2026 00:00:00 GMT for an HTTP-date).
  • A logging surface for retry attempts. Debugging rate-limit behavior without timestamped logs of attempt, delay, and jitter is effectively impossible.
  • An understanding of which HTTP statuses are retryable. 429 and 503 are throttling/transience signals. 4xx beyond 429 (401, 403, 404, 422) almost never is — retrying them just wastes the request budget.

Source: Speakeasy, “Rate Limiting Best Practices in REST API Design,” emphasizes that returning 429 with a Retry-After header is the correct server behavior because it “tells the client… they should back off for a while” and “avoids leaving the client to guess when they should try again.”

Step 1: Preparation

Capture a few constants up front. They will appear in every layer of the design.

# rate_limit_config.py
from dataclasses import dataclass

@dataclass(frozen=True)
class ClientPolicy:
    max_retries: int        # 5 — stop after this many attempts (incl. the first)
    base_delay: float       # 1.0 seconds for the first backoff
    max_delay: float        # 60.0 seconds ceiling (capped exponential backoff)
    jitter_fraction: float  # 0.5 — jitter spans 50%–100% of the computed delay
    retry_statuses: tuple   # (429, 503) — only retry these statuses

DEFAULT = ClientPolicy(
    max_retries=5,
    base_delay=1.0,
    max_delay=60.0,
    jitter_fraction=0.5,
    retry_statuses=(429, 503),
)

The max_delay cap is what AWS Builders’ Library calls capped exponential backoff: exponential functions grow so fast that without a ceiling, retry 8 alone could wait four minutes. Pair the cap with max_retries so the client also eventually gives up — a client that retries forever starves the caller of a timely error and hides upstream outages.

Step 2: Core Implementation

Layer 1 — Exponential backoff with jitter

This is the foundation every senior engineer should know by heart. The formula, in pseudo-math, is wait = min(base * 2**attempt, max_delay) * (1 - random()*jitter_fraction). The jitter term is not optional: without it, every client that hits the limit at the same instant will retry at the same instant — the thundering herd.

# backoff_client.py
import asyncio
import email.utils
import random
from datetime import datetime, timezone
import httpx

from rate_limit_config import ClientPolicy, DEFAULT


def parse_retry_after(value: str | None, now: datetime) -> float:
    """Retry-After may be seconds or an HTTP-date."""
    if not value:
        return float("nan")
    if value.strip().isdigit():
        return float(value)
    try:
        dt = email.utils.parsedate_to_datetime(value)
        return max(0.0, (dt - now).total_seconds())
    except (TypeError, ValueError):
        return float("nan")


async def request_with_backoff(
    client: httpx.AsyncClient,
    method: str,
    url: str,
    *,
    policy: ClientPolicy = DEFAULT,
    **kwargs,
) -> httpx.Response:
    """Issue an HTTP request with exponential backoff on retriable statuses."""
    for attempt in range(policy.max_retries + 1):
        response = await client.request(method, url, **kwargs)
        if response.status_code not in policy.retry_statuses:
            return response  # success or non-retriable error

        if attempt == policy.max_retries:
            response.raise_for_status()  # final attempt — surface the 429/503

        # Prefer the server's hint; fall back to computed exponential delay
        server_hint = parse_retry_after(
            response.headers.get("Retry-After"),
            datetime.now(timezone.utc),
        )
        base = server_hint if not (server_hint != server_hint) else \
               min(policy.base_delay * (2 ** attempt), policy.max_delay)
        jitter = base * (1 - random.random() * policy.jitter_fraction)
        print(f"retry attempt {attempt + 1}/{policy.max_retries} after {jitter:.2f}s")
        await asyncio.sleep(jitter)
    return response  # unreachable, defensive

Two implementation details that pay for themselves:

  1. Honor Retry-After first. A server that sends it is telling you the exact wait. Computing your own delay in that case is just guessing. The Speakeasy article makes this explicit: “Ignoring this header means you’re guessing when you could be precise.”
  2. Jitter as a multiplier, not an additive constant. A constant ±200 ms jitter makes no difference when the base delay is 32 seconds. Scaling jitter with the delay (0.5–1.0 × base) keeps the spread meaningful across the whole backoff curve, matching the recommendation in Exponential Backoff and Jitter.

Layer 2 — Client-side token bucket

Backoff alone lets you hammer the API until you receive the 429. A client-side token bucket lets you self-throttle before the request leaves the process, which is cheap (no network round trip) and frees up the upstream’s budget for genuine demand. The token bucket pattern refills tokens at a configured rate and blocks (or rejects) when the bucket is empty.

# token_bucket.py
import asyncio
import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    capacity: int          # e.g. 100 — burst size
    refill_per_sec: float  # e.g. 10.0 — steady-state rate
    _tokens: float = field(init=False)
    _last_refill: float = field(init=False)
    _lock: asyncio.Lock = field(default_factory=asyncio.Lock, init=False)

    def __post_init__(self) -> None:
        self._tokens = float(self.capacity)
        self._last_refill = time.monotonic()

    def _refill(self) -> None:
        now = time.monotonic()
        self._tokens = min(
            self.capacity,
            self._tokens + (now - self._last_refill) * self.refill_per_sec,
        )
        self._last_refill = now

    async def acquire(self, timeout: float = 30.0) -> bool:
        deadline = time.monotonic() + timeout
        while True:
            async with self._lock:
                self._refill()
                if self._tokens >= 1:
                    self._tokens -= 1
                    return True
            if time.monotonic() >= deadline:
                return False
            await asyncio.sleep(0.05)

Compose the two layers: ask the bucket for a token before the request, and apply backoff when the server still says 429. The two layers handle different failure modes — the bucket smooths bursts proactively, backoff reacts when the server disagrees.

Layer 3 — Internal rate-limit proxy

If you have multiple services calling the same upstream API, independent retry logic in each service is a recipe for correlated failure: each thinks it owns the full rate budget, and they all retry in lockstep. As Stanislav Lazarenko’s article on client-side 429 handling puts it, when services “all hit 429s simultaneously, all back off, all retry at roughly the same time, and the cycle continues”, the solution is a single internal proxy service between your services and the external API. The proxy holds the global view of rate consumption, queues requests when tokens are exhausted, and applies fair scheduling across callers. Each downstream service keeps a Client-Side Awareness layer for resilience against the proxy itself being unavailable — you don’t choose one or the other, you layer them.

Step 3: Verification and Tuning

Build the observability and guardrails before you need them.

  • Log every retry with attempt, computed delay, server-reported Retry-After, and the response status. Without this data you cannot tell whether backoff is working or whether the upstream is silently throttling every request.
  • Instrument a metric counter rate_limit_retries_total{status="<429|503>"}. If the counter is increasing smoothly, the client is adapting. If it spikes after a deployment, a config change likely tightened the limit.
  • Tune max_retries to your deadline budget. With base=1, max=60, five attempts take a worst-case 1+2+4+8+16 ≈ 31s. If your caller’s timeout is 10s, your client will be killed mid-retry — set max_retries=3 and accept the higher failure rate.
  • Set explicit request timeouts. The AWS Builders’ Library article is emphatic: every remote call needs a timeout covering connect, request, and DNS/TLS handshake. A hung connection is functionally identical to a 429 it just costs you more to detect.
  • Test idempotency before enabling retries. Backoff only works safely on idempotent endpoints. For non-idempotent writes (payments, resource creation), use an idempotency key or move retries to a higher layer that can detect duplicates.
  • Run chaos drills against the proxy. If Layer 3 is a single point of failure, it must be able to fail over. Inject 200 ms of latency into the proxy path and confirm Layer 1 still routes around it.

Best Practices Summary

  • Always honor Retry-After when present. It’s the server’s precise hint; ignoring it makes you guess at the wrong moment.
  • Jitter is not optional. Without it, exponentially-backed-off clients synchronize their retries and re-trigger the limit. Scale jitter with the delay, don’t use an additive constant.
  • Cap both the delay and the retry count. Capped exponential backoff without a retry ceiling becomes an effectively-infinite wait; retries without a cap become exponential amplification of upstream load.
  • Add a client-side token bucket before the request. Proactive self-throttling is cheaper than reactive 429s and leaves upstream budget for legitimate demand.
  • Use an internal proxy when multiple services share an upstream. It centralizes rate tracking, enables fair scheduling, and prevents each service from assuming it owns the full budget.

References