Deep Dive: Consistency & Distributed Systems (Advanced)

Theory

Advanced system design is about making correctness and recovery work under:

• partial failures
• network partitions
• retries and duplicates
• changing requirements and load patterns

1) Consistency models (practical meaning)

Choose based on what failures you can tolerate:

• Strong consistency: linearizable reads/writes
  • usually harder/less available under partitions
• Eventual consistency: replicas converge over time
  • needs compensating logic (reconciliation, versioning)

2) Quorums and replication reasoning

A common mental model:

• reads/writes route through a set of nodes
• success depends on quorum size vs fault tolerance

3) Exactly-once is a system property, not a magic switch

To get exactly-once effects (not necessarily exactly-once messages), use:

• idempotency keys / unique constraints
• deduplication tables
• transactional outbox / inbox

4) Failure-aware design patterns

Patterns to recognize and choose:

• circuit breakers
• bulkheads (isolate failures)
• fallback modes
• timeouts everywhere
• backpressure between components

Code Example (Pattern: Transactional Outbox)

Goal: reliably publish an event after a DB change without losing events or duplicating them.

-- Example tables
-- business_tx: store business write
-- outbox: store event to publish

-- 1) In a single transaction, write business data + outbox record
BEGIN;

INSERT INTO business_tx (tx_id, status, created_at)
VALUES ($1, 'COMPLETED', NOW());

INSERT INTO outbox (event_id, tx_id, event_type, payload, created_at)
VALUES ($2, $1, 'TxCompleted', $3::jsonb, NOW());

COMMIT;

-- 2) A separate publisher reads outbox and publishes
-- then marks event as delivered (or uses a unique constraint)

Practice

1. Pick a distributed feature (e.g., “order placed”).
2. List:
   • failure modes (DB down, queue delay, publisher crash, network partition)
   • which operations must be idempotent
3. Propose:
   • one consistency strategy
   • one outbox/inbox-like mechanism
   • one monitoring plan (SLIs/SLOs)

Common pitfalls

• Confusing “eventual” with “broken”
• Ignoring duplicate delivery (retries happen)
• Designing without recovery paths (what happens after a crash?)
• Treating timeouts/retries as an afterthought