Resilience and Latency
Latency and percentiles, timeouts and deadline propagation, retries with backoff and jitter, circuit breaker, bulkhead, backpressure, load shedding, and hedged requests.
9 questions
JuniorTheoryVery commonWhy does every outbound call in a Go service need a timeout, and how do you set one?
Why does every outbound call in a Go service need a timeout, and how do you set one?
Without a timeout a slow or hung dependency pins a goroutine and its connection forever, so under load those leaks pile up until the service exhausts them and fails. You bound each call with ctx, cancel := context.WithTimeout(parent, budget), pass ctx in, and watch ctx.Done(), so it aborts when the budget expires.
Common mistakes
- ✗Omitting timeouts on outbound calls so a hung dependency silently leaks goroutines and connections under load
- ✗Setting a single global client timeout instead of a per-call deadline scoped to each request's budget
- ✗Calling
context.WithTimeoutbut never checkingctx.Done()or passingctxinto the actual call
Follow-up questions
- →How do you propagate the remaining deadline to downstream calls in the chain?
- →What is the difference between
context.WithTimeoutandcontext.WithDeadline?
MiddleTheoryVery commonHow does the resilience pattern circuit breaker work, and what failure does it solve?
How does the resilience pattern circuit breaker work, and what failure does it solve?
It tracks closed -> open -> half-open states. When the error rate crosses a threshold it trips open and fails fast without calling the broken dependency. After a cool-down it lets one probe through (half-open) to test recovery, closing again on success. This stops cascading failure and gives the dependency room to recover.
Common mistakes
- ✗Thinking the breaker retries the call — it does the opposite: it fails fast and skips the dependency entirely while open.
- ✗Forgetting the
half-openprobe, so the breaker never re-tests recovery and stays open even after the dependency heals. - ✗Setting the threshold on raw error count instead of error rate, so a busy service trips on noise during normal traffic.
Follow-up questions
- →How do you tune the error threshold and the open-state cool-down window?
- →Why pair the resilience pattern circuit breaker with per-call timeouts and deadlines?
MiddleCodeVery commonHow do you write a retry helper retrying only transient errors with backoff plus jitter that honors ctx?
How do you write a retry helper retrying only transient errors with backoff plus jitter that honors ctx?
Loop up to a fixed attempt cap calling op; on success return nil, retry only on a transient error, else return it. Between attempts sleep base*2^n (1s, 2s, 4s) plus random jitter, and select on ctx.Done() so a cancelled ctx aborts mid-wait.
Common mistakes
- ✗Retrying non-idempotent or non-transient errors, duplicating side effects or hammering a permanent failure
- ✗Backing off without jitter, so all clients retry in lockstep and create a synchronized thundering herd
- ✗Sleeping with a bare
time.Sleepthat ignoresctx, so a cancelled call keeps waiting out the full backoff
Follow-up questions
- →Why add jitter instead of a fixed offset between retries?
- →How does a retry budget prevent retry amplification across a chain?
JuniorTheoryCommonWhy measure request latency with p50/p95/p99 percentiles instead of the average?
Why measure request latency with p50/p95/p99 percentiles instead of the average?
An average hides the tail: a low mean can sit beside a terrible p99 many real users actually hit. Percentiles describe the worst case, which dominates under fan-out — a request to N services is only as fast as its slowest dependency.
Common mistakes
- ✗Reporting only the mean latency and declaring the service fast while a brutal p99 quietly hurts a large slice of users.
- ✗Forgetting fan-out: assuming a request is as fast as its average dependency, when it is gated by the single slowest one.
Follow-up questions
- →How does fan-out across many services amplify a single dependency tail?
- →How do latency, throughput and concurrency relate via the queueing law Little's law?
JuniorTheoryCommonWhat is load shedding and why should an overloaded service do it?
What is load shedding and why should an overloaded service do it?
Under overload you deliberately drop low-priority or excess work (admission control) to protect the core path, returning 503/429 fast instead of dying slowly. Shed early, before collapse, so the requests you do accept still succeed.
Common mistakes
- ✗Confusing load shedding with backpressure — shedding drops work, backpressure tells the upstream to slow down.
- ✗Shedding too late, only after queues are full and latency has already collapsed for everyone.
- ✗Dropping requests silently or with a 500 instead of a fast 503/429 the client can retry sensibly.
Follow-up questions
- →How do you decide which requests to shed first under overload?
- →What status code do you return and why does it matter for the caller?
MiddleTheoryCommonHow do the isolation pattern bulkhead and backpressure keep one overloaded dependency from sinking a service?
How do the isolation pattern bulkhead and backpressure keep one overloaded dependency from sinking a service?
A bulkhead isolates resources per dependency — separate goroutine or connection pools, bounded queues — so one saturated dependency can't exhaust ALL of them. Backpressure signals upstream to slow down via bounded channels; a full buffer blocks or sheds, never grows.
Common mistakes
- ✗Sharing one global goroutine or connection pool across all dependencies, so a single slow one starves everyone
- ✗Buffering without bound under load instead of blocking or shedding, which just delays the crash into an OOM
- ✗Confusing a bulkhead (isolation) with a circuit breaker (fail-fast) — they solve different failure shapes
Follow-up questions
- →Where do you put the bounded queue and how do you size its capacity?
- →How does backpressure interact with the technique load-shedding under sustained overload?
MiddleTheoryOccasionalHow do request coalescing and hedged requests each affect tail latency and load?
How do request coalescing and hedged requests each affect tail latency and load?
Coalescing dedupes concurrent identical in-flight requests into ONE call (singleflight), killing a thundering herd / cache stampede on a hot key — it cuts load, not a single request's latency. Hedged requests cut the tail: after ~p95, send a second copy to another replica, take whichever wins (cancel the loser), at extra-load cost, so cap the hedge rate (~<=5%).
Common mistakes
- ✗Thinking coalescing lowers a single request's latency — it only cuts duplicate load; the one in-flight call is no faster.
- ✗Hedging everything: an uncapped hedge rate doubles load and can tip a struggling dependency over instead of saving the tail.
- ✗Firing the hedge immediately instead of after ~p95, so most requests get a needless second copy with no tail benefit.
Follow-up questions
- →Why does
singleflightcoalesce only within one process, not across instances? - →How do tied requests differ from plain hedged requests in cancellation?
SeniorDesignOccasionalA read endpoint fans out to ~8 downstream services per request; p50 is fine but p99 is unacceptable, and you cannot change the downstreams, must stay inside a fixed latency budget, and must not amplify their load. Design how you would tame the tail.
A read endpoint fans out to ~8 downstream services per request; p50 is fine but p99 is unacceptable, and you cannot change the downstreams, must stay inside a fixed latency budget, and must not amplify their load. Design how you would tame the tail.
First find the offender: under fan-out the slowest of the 8 dependencies dominates the tail, so measure per-dependency p99. Derive a per-call deadline from the request budget and propagate it via context so no hop resets it. Hedge slow replicas after ~p95 with a capped hedge rate (e.g. <=5%), coalesce duplicate hot-key reads into one call, and avoid unbounded retries that amplify load.
Common mistakes
- ✗Tuning the aggregate
p99instead of per-dependency percentiles, so the actual slow offender stays hidden behind the others. - ✗Resetting the deadline on each downstream hop, which lets the total time blow past the request budget under fan-out.
- ✗Hedging every call with no rate cap, which doubles the load on the downstreams the constraints said you must not amplify.
Follow-up questions
- →How do you size the per-call deadline when the 8 calls run concurrently versus serially?
- →What signal shows the tail-cutting technique of hedged requests helps rather than just adds wasted load?
SeniorDesignOccasionalA Go service hits a sudden 10x traffic spike from a viral event while one downstream dependency is simultaneously degrading; with fixed downstream capacity, no time to add machines, and a requirement to keep the core path alive, how do you design the overload controls so the service degrades gracefully and stays up instead of collapsing?
A Go service hits a sudden 10x traffic spike from a viral event while one downstream dependency is simultaneously degrading; with fixed downstream capacity, no time to add machines, and a requirement to keep the core path alive, how do you design the overload controls so the service degrades gracefully and stays up instead of collapsing?
Layer overload controls: bounded queues apply backpressure, load-shedding drops low-priority work with fast 503/429 so accepted requests finish, per-dependency bulkhead isolation contains the degrading one, and a circuit breaker fails fast on it — so the service degrades gracefully instead of collapsing into unbounded queues and OOM.
Common mistakes
- ✗Letting queues grow unbounded to avoid rejecting requests, which just defers collapse into OOM and worse tail latency.
- ✗Retrying the degrading dependency aggressively, amplifying its load and turning a partial outage into a full cascade.
- ✗Treating load shedding as failure rather than as protection for the accepted requests on the core path.
Follow-up questions
- →How would you pick which requests to shed first under overload?
- →Where do you set the bounded-queue size, and how does that bound tail latency?
