Rate limits from both sides: handling 429 responses and limiting your own API

Sooner or later you call an API that says no. Sooner or later you run an API and have to say no. Rate limits are the polite way to do it, and they work best when both sides understand the conversation.

What a 429 says

The status code 429 Too Many Requests comes from RFC 6585. It means the user has sent too many requests in a given amount of time. The response may include a Retry-After header telling the client how long to wait. The RFC deliberately leaves out how a server counts requests or identifies a user: per resource, per server, per credential or per cookie are all allowed.

Two practical consequences follow. A 429 is a normal, expected answer, not a crash. And the details of the limit are specific to each service, so read its documentation.

Handling 429 as a client

Stripe's rate limit documentation is a good model of what a provider asks of you: watch for 429 responses, retry with exponential backoff, and add randomness so that many clients do not retry in lockstep, the thundering herd problem.

A small retry loop captures it:

import random, time

def call_with_retry(send, max_tries=6):
    delay = 0.5
    for attempt in range(max_tries):
        resp = send()
        if resp.status_code != 429:
            return resp
        wait = float(resp.headers.get("Retry-After", 0)) or delay
        time.sleep(wait + random.uniform(0, wait / 2))
        delay = min(delay * 2, 30)
    raise RuntimeError("still rate limited")

Points worth copying:

• Honour Retry-After when it is present; it is better information than your own guess.
• Cap the delay and the number of tries, so a stuck call fails visibly.
• Only retry requests that are safe to repeat. For writes, send an idempotency key so a retry cannot create a second charge or record.

Not every limit is a request rate

Stripe's page separates two kinds of limit. A rate limit counts requests over time. A concurrency limit counts requests in progress at one moment, which you hit when you send many slow requests at once. The fix differs: slow down for the first, send fewer in parallel for the second. Stripe also returns a header naming the reason for the limit, which makes it possible to tell them apart in code.

The same page notes that a 429 without those headers may not be a rate limit at all but a lock timeout, where two requests touched the same object. The advice there is to queue changes to one object and run them in sequence. The lesson for your own logs: record the headers with each 429 so you can tell which kind you are looking at.

Smoothing your own traffic

Retrying after a rejection works, but it is better not to be rejected. Stripe suggests a client-side token bucket: a bucket holds up to N tokens, refills at a steady rate, and each request spends one. When the bucket is empty, the caller waits.

import time

class TokenBucket:
    def __init__(self, rate, burst):
        self.rate, self.capacity = rate, burst
        self.tokens, self.last = burst, time.monotonic()

    def take(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return 0.0
        return (1 - self.tokens) / self.rate # seconds to wait

A bucket allows short bursts up to its capacity while holding the long-run rate steady, which is how most real traffic behaves. If several processes share a limit, the bucket has to be shared too, for example in a database row or a cache, or each process should get its share of the budget.

flowchart LR
  A[Requests] --> B{Token available?}
  B -- yes --> C[Send]
  B -- no --> D[Wait or queue]
  D --> B

Limiting your own API

When you are the one saying no, some habits make you easy to work with:

1. Return 429 with Retry-After. Clients that follow the standard will behave well without special code.
2. Say which limit was hit. A short response body, or a header like Stripe's, saves support questions.
3. Choose the key deliberately. Per account is usually fairer than per IP address, which punishes people behind shared networks.
4. Limit expensive endpoints separately. A search or export call can cost far more than a lookup; a single global number hides that. Stripe lists different limits per endpoint for the same reason.
5. Document the numbers and the behaviour. Treat limits as part of the contract.
6. Do not cache 429 responses. RFC 6585 says they must not be stored by a cache.

Note

Rate limiting protects your service from overload and one noisy client from everyone else. It is not a security boundary on its own. Pair it with authentication and the least-privilege thinking in giving an AI agent only the access it needs, which applies to software clients of any kind.

Calling model APIs and agents

The same rules apply to language model APIs, where limits are often counted in tokens as well as requests, and where an agent loop can generate bursts without anyone noticing. Put a shared token bucket in front of the model client, log every 429, and let the loop back off instead of failing the whole task.

A short checklist

• Retry only safe requests, with capped exponential backoff and jitter.
• Respect Retry-After.
• Throttle on the client before the server has to.
• Log the reason for each limit.
• On your side, return clear 429 responses and publish your limits.

Sources

• RFC 6585, section 4: 429 Too Many Requests
• Stripe API documentation: Rate limits