Consistency and Distributed Transactions
The CAP theorem, consistency models, two- and three-phase commit, the TCC and SAGA patterns, and idempotency of distributed operations.
8 questions
JuniorTheoryVery commonWhat does the CAP theorem actually say a distributed system must trade off?
What does the CAP theorem actually say a distributed system must trade off?
CAP says that during a network PARTITION you must choose between Consistency and Availability — be CP (block or refuse, staying correct) or AP (keep serving, risk stale data). It is behavior UNDER a partition, not a steady-state "pick 2 of 3"; you tune the C-vs-A trade-off per operation.
Common mistakes
- ✗Treating CAP as a steady-state "pick 2 of 3" instead of a choice that arises only during a partition
- ✗Calling a system CA, as if you could keep both C and A while a partition is actually happening
- ✗Forgetting that the C-vs-A trade-off is tuned per operation, not fixed once for the whole system
Follow-up questions
- →During a partition, when would a payment service deliberately choose CP over AP?
- →How do consistency models like read-your-writes fit between strong and eventual?
JuniorTheoryVery commonWhat is the difference between strong and eventual consistency, and what does each one cost you?
What is the difference between strong and eventual consistency, and what does each one cost you?
Strong consistency means every read sees the latest write immediately, as if a single copy existed (linearizable). Eventual consistency lets replicas diverge briefly but converge once writes stop. Strong costs latency and availability; eventual buys availability and scale.
Common mistakes
- ✗Treating eventual consistency as 'broken' rather than a deliberate trade for availability and scale
- ✗Assuming strong consistency is free and forgetting it costs latency and availability
- ✗Confusing the two definitions — claiming eventual reads always see the latest write
Follow-up questions
- →Where do read-your-writes and monotonic reads fit on this spectrum?
- →How does the theorem CAP (consistency-availability-partition) relate to this choice?
JuniorTheoryCommonHow does two-phase commit (2PC) coordinate a transaction across services?
How does two-phase commit (2PC) coordinate a transaction across services?
A coordinator sends PREPARE to every participant; each votes yes or no. It sends COMMIT only if all voted yes, otherwise ABORT. The lock-free alternative is the SAGA pattern — a chain of local transactions, each with a compensating action.
Common mistakes
- ✗Thinking 2PC commits without a vote round — the
PREPAREphase exists precisely to gather every participant's yes/no first. - ✗Believing 2PC is non-blocking — if the coordinator dies after
PREPARE, participants hang holding locks until it returns. - ✗Confusing
SAGAwith 2PC —SAGAhas no global commit; it undoes work with compensating transactions, not a rollback.
Follow-up questions
- →Why is 2PC called a blocking protocol, and what failure causes the block?
- →When would you pick the pattern
SAGAover 2PC for a cross-service workflow?
MiddleDesignCommonAn e-commerce order flow spans four services with their own databases — billing charges the card, inventory reserves stock, orders persists the order, notifications emails the buyer — and each is a separate network call. You need the flow to stay consistent without holding a global lock across services, and inventory frequently fails mid-flow when stock runs out, after billing has already charged the card. Calls can be retried and may run twice. Design how to keep the data correct across all four services: how you model each step and its undo, what happens (in which order) when inventory fails after billing succeeds, how you coordinate the steps, how retried operations stay safe, and how events are published reliably. What consistency guarantee does this give?
An e-commerce order flow spans four services with their own databases — billing charges the card, inventory reserves stock, orders persists the order, notifications emails the buyer — and each is a separate network call. You need the flow to stay consistent without holding a global lock across services, and inventory frequently fails mid-flow when stock runs out, after billing has already charged the card. Calls can be retried and may run twice. Design how to keep the data correct across all four services: how you model each step and its undo, what happens (in which order) when inventory fails after billing succeeds, how you coordinate the steps, how retried operations stay safe, and how events are published reliably. What consistency guarantee does this give?
Model the flow as a SAGA: each step (billing, then inventory, then orders, then notifications) is a local transaction with a compensating action. If inventory fails, run compensations backward — refund the charge — so no global lock is held. Drive it with an orchestrator, make every step and compensation idempotent via an idempotency key, and publish events through an outbox, yielding eventual consistency.
Common mistakes
- ✗Reaching for two-phase commit and a global lock across services instead of local transactions with compensations
- ✗Forgetting compensations run backward — refunding billing when inventory fails after the charge
- ✗Skipping idempotency keys, so a retried compensation refunds or releases stock twice
Follow-up questions
- →Orchestration or choreography here, and what does each cost you operationally?
- →Why does the outbox pattern beat publishing the event inside the same transaction?
MiddleTheoryOccasionalWhy is two-phase commit (2PC) called a blocking protocol when the coordinator dies after the prepare phase?
Why is two-phase commit (2PC) called a blocking protocol when the coordinator dies after the prepare phase?
Participants cannot resolve a vote alone. After PREPARE they hold their locks awaiting the coordinator's COMMIT or ABORT; if it dies in that window they hang with locks held, blocking other transactions. The coordinator also waits for the slowest participant, so tail latency tracks it.
Common mistakes
- ✗Thinking a coordinator crash after PREPARE is harmless because the votes were already collected
- ✗Believing participants can unilaterally commit or abort once they have voted yes
- ✗Assuming 2PC is non-blocking and ignoring that locks are held for the whole round trip
Follow-up questions
- →How does three-phase commit (3PC) make this non-blocking, and what does it cost?
- →Why does longer lock-holding under 2PC worsen transaction isolation contention?
MiddleTheoryOccasionalHow does the distributed-transaction pattern try-confirm-cancel (TCC) work?
How does the distributed-transaction pattern try-confirm-cancel (TCC) work?
The coordinator calls try on each service to reserve the resource (e.g. put a hold on funds); if all succeed it calls confirm on all to finalize, otherwise cancel on all to release. Each service exposes try/confirm/cancel, persists reservation state, and must be idempotent so retries are safe.
Common mistakes
- ✗Confusing TCC with
2PC— TCC has no global lock and uses a reserve step instead of a vote-then-commit - ✗Forgetting that each service must persist reservation state so
confirm/cancelknow what to finalize - ✗Skipping idempotency, so a retried
confirmorcanceldouble-applies the change
Follow-up questions
- →Why must
try/confirm/cancelbe idempotent in TCC? - →How is TCC different from the orchestration pattern saga with compensating transactions?
SeniorDesignOccasionalFor a payment-and-inventory checkout flow spanning a billing service and an inventory service, you must avoid the blocking locks of distributed-commit protocol 2PC (two-phase commit). Business requires that funds and stock be held atomically the moment checkout starts, no other order may consume that held stock, and a partial failure must leave no charge and no stock decrement. Choose between the application-transaction pattern TCC (try-confirm-cancel) and the long-transaction pattern SAGA, and justify which one fits these isolation and reservation constraints.
For a payment-and-inventory checkout flow spanning a billing service and an inventory service, you must avoid the blocking locks of distributed-commit protocol 2PC (two-phase commit). Business requires that funds and stock be held atomically the moment checkout starts, no other order may consume that held stock, and a partial failure must leave no charge and no stock decrement. Choose between the application-transaction pattern TCC (try-confirm-cancel) and the long-transaction pattern SAGA, and justify which one fits these isolation and reservation constraints.
Pick TCC: its try phase reserves funds and stock up front (a hold), so no concurrent order sees that stock, then confirm finalizes the reservation or cancel releases it — near-isolation in two round trips, at the cost that every service must support try/confirm/cancel idempotently. SAGA instead commits each local step then compensates, exposing a charged-but-not-reserved intermediate state that violates the hold requirement. Both avoid 2PC's blocking locks.
Common mistakes
- ✗Calling TCC and SAGA interchangeable — only TCC's
tryphase gives an up-front reservation hold. - ✗Believing SAGA hides the intermediate state; its committed-then-compensated steps are externally visible.
- ✗Thinking TCC keeps 2PC-style DB locks open; it holds an application reservation, not a database lock.
Follow-up questions
- →How do you keep TCC's
confirmandcancelidempotent under retries? - →What happens to a SAGA if a compensating action itself fails partway?
MiddleTheoryRareWhy does three-phase commit add a PRE-COMMIT phase to 2PC (two-phase commit), and why is it rarely used in practice?
Why does three-phase commit add a PRE-COMMIT phase to 2PC (two-phase commit), and why is it rarely used in practice?
3PC inserts a PRE-COMMIT phase between the vote and the commit. Once a participant sees PRE-COMMIT it knows everyone voted yes, so it can commit on its own if the coordinator dies — it never blocks holding locks like 2PC. It is rarely used: the extra round trip costs latency and it stays correct only on a synchronous network.
Common mistakes
- ✗Calling 3PC a drop-in upgrade for 2PC, ignoring that its non-blocking guarantee assumes a synchronous network and breaks under partitions.
- ✗Saying 3PC eliminates blocking entirely, when it only avoids blocking on coordinator failure and still pays an extra round trip.
- ✗Confusing PRE-COMMIT with actually writing the data, rather than an agreement that all participants voted yes.
Follow-up questions
- →Why does 3PC's non-blocking guarantee fail on an asynchronous network with partitions?
- →How would the consensus protocols Paxos or Raft solve the same atomic-commit problem differently?
