Scaling and Load Balancing
Vertical versus horizontal scaling, the scalability laws of Amdahl, Gustafson, and USL, load-balancing algorithms, session affinity, and service discovery.
8 questions
JuniorTheoryVery commonWhat is the difference between vertical and horizontal scaling, and which do you reach for first?
What is the difference between vertical and horizontal scaling, and which do you reach for first?
Vertical scaling means a bigger box — simple, but with a hard ceiling and single point of failure. Horizontal means more instances behind a load balancer: needs stateless instances and harder consistency. Squeeze vertical first, go horizontal when one box can't cope.
Common mistakes
- ✗Swapping the definitions — calling 'more machines' vertical and 'a bigger box' horizontal
- ✗Forgetting that vertical scaling has a hard ceiling and stays a single point of failure
- ✗Going horizontal first while instances still hold local state, so requests break on failover
Follow-up questions
- →What has to be true about an instance before you can scale it horizontally?
- →Why does horizontal scaling make consistency harder than a single box did?
JuniorTheoryCommonWhat are the common load-balancing algorithms, and when does each one fit?
What are the common load-balancing algorithms, and when does each one fit?
Round-robin rotates evenly across replicas; weighted round-robin is capacity-aware; least-connections sends to the replica with the fewest open connections; response-time-based picks the lowest-latency replica; consistent hashing keys a request sticky to one replica. L4 balances by transport, L7 is HTTP-aware.
Common mistakes
- ✗Naming round-robin as the only algorithm and ignoring capacity- or latency-aware choices like weighted or least-connections.
- ✗Confusing the layers: thinking L4 is HTTP-aware when L4 balances by transport and only L7 reads HTTP.
- ✗Assuming consistent hashing just spreads load evenly, missing that its point is sticky routing of a key to one replica.
Follow-up questions
- →When would you pick the balancing algorithm consistent hashing over least-connections?
- →Why does response-time-based routing need health and latency probes?
JuniorTheoryCommonWhy are stateless sessions preferred over sticky sessions when scaling a service horizontally?
Why are stateless sessions preferred over sticky sessions when scaling a service horizontally?
Stateless sessions keep state in a JWT or a shared Redis store, so any instance can serve any user and the balancer spreads load evenly. Sticky sessions pin a user to one instance, skewing load and breaking on failover.
Common mistakes
- ✗Thinking sticky sessions are needed for login — a shared session store or
JWTkeeps users logged in across any instance. - ✗Believing ip-based affinity balances load; a few large clients behind one NAT can pin most traffic to a single instance.
Follow-up questions
- →How does the token format
JWTlet any instance authenticate a request without a shared store? - →When would you still accept sticky sessions despite the load skew?
MiddleTheoryCommonWhat do Amdahl's, Gustafson's, and the universal scalability law (USL) tell you about adding nodes?
What do Amdahl's, Gustafson's, and the universal scalability law (USL) tell you about adding nodes?
Amdahl's law bounds speedup by the serial fraction, so extra cores give diminishing returns. Gustafson's law says speedup stays near-linear if you grow the problem with the resources. The scalability law USL adds a coherency penalty: throughput rises, plateaus, then drops.
Common mistakes
- ✗Confusing Amdahl's law with Gustafson's law — treating the serial fraction as fixed when the problem itself grows
- ✗Assuming more nodes always raises throughput, ignoring the USL coherency penalty that makes it plateau and then decline
- ✗Reading near-linear speedup as a promise of free scaling, forgetting the coordination cost of every added node
Follow-up questions
- →When would Gustafson's law apply but Amdahl's pessimistic bound not?
- →What kinds of coordination drive the USL coherency penalty in a service?
MiddleTheoryCommonHow does service discovery route traffic to live instances, and how do client-side and server-side differ?
How does service discovery route traffic to live instances, and how do client-side and server-side differ?
A registry — Consul, etcd, Kubernetes DNS — maps a service name to its healthy instances; health-aware discovery drops dead ones so the balancer skips them. Client-side has the caller query the registry; server-side hides it behind a balancer.
Common mistakes
- ✗Treating the registry as a static address list and forgetting health checks prune failing instances
- ✗Confusing the directions — calling client-side discovery server-side, or putting the balancing in the wrong place
- ✗Assuming a crashed instance keeps getting traffic, ignoring that health-aware discovery removes it from lookups
Follow-up questions
- →How does the registry decide an instance is healthy enough to receive traffic?
- →What goes wrong if the registry itself becomes a single point of failure?
MiddleDesignOccasionalYou must ship a new version of a read-heavy Go API to production with stateless sessions, only a small traffic fraction at first, error-rate and latency monitoring, and fast rollback; how do you roll it out?
You must ship a new version of a read-heavy Go API to production with stateless sessions, only a small traffic fraction at first, error-rate and latency monitoring, and fast rollback; how do you roll it out?
Do a canary: route a small percentage of traffic to the new version via a feature-flag-driven split, keep instances stateless (JWT or shared Redis) so either version serves any user, watch error rate and latency, then roll forward or back. Sticky sessions would pin users and break the split.
Common mistakes
- ✗Treating sticky sessions as compatible with a canary — affinity pins users to one version and defeats the percentage split.
- ✗Big-bang deploying everything at once with no traffic fraction and no fast rollback path.
- ✗Forgetting to watch error rate and latency on the canary, so a bad version is promoted blind.
Follow-up questions
- →How does the rollout pattern blue-green differ from a canary rollout?
- →Where does the rollout toggle feature flag live so the split applies instantly?
SeniorDesignOccasionalA single-box read-heavy Go API must absorb roughly 10x its current read traffic. Constraints: keep downtime minimal, keep instances replaceable, balance load fairly across replicas, and roll the change out gradually so a bad release can be caught early. How do you scale the service out, distribute traffic across the new replicas, and introduce the added capacity safely?
A single-box read-heavy Go API must absorb roughly 10x its current read traffic. Constraints: keep downtime minimal, keep instances replaceable, balance load fairly across replicas, and roll the change out gradually so a bad release can be caught early. How do you scale the service out, distribute traffic across the new replicas, and introduce the added capacity safely?
Make instances stateless and scale out horizontally behind an L7 load balancer with least-connections or latency-aware routing. Register replicas in service discovery with health checks so dead ones drop, then roll new capacity in via a canary traffic split before full cutover.
Common mistakes
- ✗Going vertical only and hitting the hard ceiling, leaving one box as a single point of failure under 10x load
- ✗Keeping session state on the instance, so sticky routing is forced and replicas cannot be swapped freely
- ✗Cutting all traffic to the new fleet at once with no canary, so a bad release takes down every read request
Follow-up questions
- →Which balancing algorithm fits replicas with uneven latency, and why not plain round-robin?
- →What signals decide when to widen a canary rollout versus roll the traffic split back?
SeniorTheoryRareWhy can adding instances behind the service-discovery registry slow a system, and how do you diagnose it?
Why can adding instances behind the service-discovery registry slow a system, and how do you diagnose it?
The USL coherency term: instances that must coordinate (shared cache, lock, registry chatter) pay a crosstalk cost that grows faster than linear, so throughput plateaus then declines. Diagnose by plotting throughput against node count — a falling curve means coordination dominates. Fix it by cutting coordination (shard state), not adding nodes.
Common mistakes
- ✗Treating scale-out as free — assuming N nodes give N× throughput and ignoring the coherency penalty in the USL curve.
- ✗Reacting to a plateau by adding more instances, which worsens crosstalk instead of relieving it.
- ✗Forgetting that service discovery and health checks are themselves coordination traffic that grows with node count.
Follow-up questions
- →Where does the universal scalability law (USL) coherency curve peak for a given coordination fraction?
- →How does sharding shared state move the throughput plateau outward?
