Timeouts and Retries: How to Set Them Without Making Outages Worse

Timeouts and retries are usually configured independently, by different people, at different layers, with values chosen because they seemed reasonable. The failures they produce are not independent, and the worst ones are caused by the retry logic rather than by the original fault.

For a separate people-operations perspective, this page covers clock-in and clock-out workflows.

The mechanism is worth understanding before the settings.

What a retry does to a struggling service

A service is at capacity and responses slow down. Clients time out and retry. The retry arrives at a service that is already overloaded, and it arrives in addition to the original request, which the service is frequently still processing.

Load does not stay flat under this. With a retry policy of three attempts, a service under stress can see up to three times its normal request rate at exactly the moment it has least capacity. The original request is not cancelled by the client giving up — the server keeps working on it, and now competes with the retry.

This is why a dependency that would have recovered from a brief slowdown instead falls over, and why the outage outlasts the original cause. The system has a positive feedback loop, and retries are the gain.

A retry is only safe when the system has spare capacity. That is precisely when you least need it.

The three things a timeout must be shorter than

A timeout is not a property of one call. It sits in a chain, and the constraints come from above and below.

Shorter than the caller's timeout. If your service has a 30-second timeout on a downstream call, and your caller gives up on you after 10 seconds, the last 20 seconds of work is wasted — you are holding a connection and a thread to produce a response nobody will read.

Shorter than the resource you are holding. A request holding a database connection from a pool of 20 for 60 seconds means 20 slow requests exhaust the pool and every other request fails, including ones that had nothing to do with the slow dependency.

Longer than the realistic worst case for a healthy call. A timeout below the p99 latency of a working dependency converts normal variance into errors, and then the retries add load, and you have manufactured the outage.

The practical approach: budget from the outside in. The user-facing request has a deadline. Each layer gets a slice, and each layer passes the remaining budget down rather than starting a fresh timer. Most frameworks support deadline propagation; where they do not, passing a deadline in a header and honouring it is a small amount of code with a large effect.

Retry only what is safe to retry

Only idempotent operations. A retried POST that creates an order may create two orders. If the operation is not naturally idempotent, make it so with an idempotency key the server deduplicates on — see why every request will be retried.

Only on the right errors. A connection failure or a 503 is worth retrying. A 400 will fail identically every time. Retrying deterministic failures is pure waste and pure added load.

Never on a timeout you cannot interpret. A timeout means you do not know whether the operation happened. Retrying is a decision that duplicate execution is preferable to no execution, and for a payment it may not be.

Retry at one layer. This is the one people get wrong most expensively. If the HTTP client retries three times, the service-level policy retries three times, and the queue redelivers three times, a single failure produces 27 attempts. Retry budgets at each layer multiply.

Pick the layer deliberately — usually the one that understands the semantics — and make the others fail fast.

Backoff, and why jitter is not optional

Fixed-interval retries synchronise. A hundred clients that fail at the same moment and retry after exactly one second produce a spike at exactly one second, then at two, then at four. The dependency gets hit with coordinated waves.

Exponential backoff spreads attempts out over time, and on its own does not fix synchronisation — it just spaces the waves further apart.

Jitter breaks the synchronisation. Randomising the delay is what turns a wave into a spread. Full jitter — a random value between zero and the current backoff ceiling — is the common recommendation and is simple to implement:

delay = random.uniform(0, min(cap, base * 2 ** attempt))

Cap the total, not just the interval. A retry policy should have a deadline, not only an attempt count. Three attempts with exponential backoff can take longer than the caller is willing to wait, at which point every attempt after the first was wasted work.

Circuit breakers: the part retries cannot do

Backoff limits the rate of retries from one client. It does nothing about the aggregate when there are thousands of clients.

A circuit breaker tracks the failure rate to a dependency and, past a threshold, stops sending requests entirely for a period. Calls fail immediately without touching the network.

Two things this achieves:

The failing dependency gets a chance to recover rather than being held at saturation by traffic it cannot serve.

The caller stops burning threads and connections on calls that will fail, which is what prevents the failure spreading upward.

The half-open state is the important part. After the timeout, let a small number of requests through. If they succeed, close the circuit. If they fail, open it again. Without this, the breaker either never recovers or slams the recovering service with full traffic the moment it comes back.

Consider load shedding as well. A service that is over capacity is better off rejecting a fraction of requests immediately than accepting all of them and serving all of them slowly — a fast rejection lets the caller fail over or degrade, while a slow success holds resources at both ends.

The failure modes to recognise

Retry storm. A brief blip becomes a sustained outage because retry traffic keeps the dependency saturated. Recognisable by request rate rising as error rate rises.

Thread pool exhaustion from a missing timeout. One slow dependency stalls unrelated requests. Recognisable by everything failing when only one thing is broken.

Multiplied retries. Total attempts far exceed what any single policy specifies. Count the layers.

Retry after the caller gave up. Work continues on a request nobody is waiting for. Visible as a gap between server-side and client-side success rates.

Synchronised recovery. The dependency comes back, all clients retry simultaneously, it falls over again. This is what jitter and half-open circuits prevent.

Settings that are defensible

Not universal numbers — a way of arriving at them.

Start from the user-facing deadline. What is the maximum acceptable time for the whole request?

Subtract, do not add. Allocate that budget across the call chain, leaving headroom. Each layer passes the remainder down.

Set each timeout from measured latency, not from a round number. Something above p99 for a healthy dependency, with margin.

Retries: two attempts beyond the first, at most, for most things. Three is already aggressive.

Always jitter.

Always a deadline on the whole retry sequence, not just an attempt count.

A circuit breaker on every network dependency that is not trivially cheap.

Then verify it. Inject latency into a dependency in a staging environment and watch what your system does. Almost every configuration described above looks correct on paper and behaves differently under a slow — rather than failing — dependency. Slow is harder than down, and it is the case nobody tests.

The short version

Timeouts prevent one slow thing from consuming resources everywhere. Set them from the deadline down, and always shorter than your caller's.

Retries cover transient faults and amplify persistent ones. Only idempotent operations, only retryable errors, one layer, jittered backoff, bounded by a deadline.

Circuit breakers do the thing retries cannot — reduce aggregate load on something that is already failing.

Get the first two wrong together and the retry logic becomes the outage.