Scaling & Reliability Tradeoffs (Intermediate)

Theory

Intermediate system design means designing for failure and growth simultaneously.

1) Scaling patterns (what to use and when)

Common tools:

• Caching: reduce repeated reads
• Rate limiting: protect downstreams and control load
• Queues: buffer bursts and enable async processing
• Load balancing: distribute traffic across stateless services
• Sharding/partitioning: distribute data and traffic at higher scale

Design question:

• Is the bottleneck compute, database, network, or downstream dependency?

2) Reliability: model failure modes

Think in scenarios:

• timeouts talking to service A
• database becomes slow
• third-party API returns 500/429
• partial failures (some components succeed, others fail)

Reliability mechanisms:

• timeouts
• retries with backoff + jitter
• circuit breakers
• bulkheads
• graceful degradation

3) SLIs/SLOs: make reliability measurable

• SLI (Service Level Indicator): what you measure (latency p95, error rate)
• SLO: target (p95 < 300ms, error rate < 0.1%)
• Burn rate: how quickly you’re burning budget when things degrade

Code Example (Pseudo: retry with idempotency key)

type Request = { idempotencyKey?: string; payload: unknown };

async function callWithRetry<T>(fn: () => Promise<T>, req: Request): Promise<T> {
  const maxAttempts = 4;

  // Idempotency: send stable key so safe retries don’t create duplicates
  if (!req.idempotencyKey) req.idempotencyKey = crypto.randomUUID();

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err: any) {
      const retryable =
        err?.status === 429 ||
        err?.status === 500 ||
        err?.code === "ETIMEDOUT";

      if (!retryable || attempt === maxAttempts) throw err;

      const backoffMs = Math.min(5000, 100 * 2 ** (attempt - 1));
      const jitterMs = Math.floor(Math.random() * 100);
      await new Promise(r => setTimeout(r, backoffMs + jitterMs));
    }
  }

  throw new Error("unreachable");
}

Practice

1. Choose a system (e.g., “file upload service”).
2. Identify:
   • 2 scaling bottlenecks
   • 3 failure modes
3. Propose:
   • one caching strategy
   • one queue/async strategy
   • one retry strategy with idempotency

Common pitfalls

• Retrying non-idempotent operations
• No timeouts (requests hang forever)
• Scaling the wrong layer (DB queries vs API CPU vs network)