Incident Management & Post-Mortems

Incidents are inevitable in production systems. What separates high-performing teams is not fewer incidents—it is how quickly they detect, respond, and learn from them.

Learning outcomes

By the end you can:

• define and track SLOs and error budgets
• follow a structured incident response process
• write a blameless post-mortem
• implement reliability improvements from post-mortem action items

1) SLOs and error budgets

An SLO (Service Level Objective) defines your reliability target:

• 99.9% success rate on /api/orders over a 30-day rolling window
• p95 latency below 500ms for 95% of time

An error budget is the allowed unreliability:

• 99.9% SLO → 0.1% budget → ~43.8 minutes of downtime per month

Track error budget consumption:

# Error budget remaining (as a fraction of 1.0)
1 - (
  sum_over_time(http_requests_total{status=~"5.."}[30d])
    / sum_over_time(http_requests_total[30d])
) / (1 - 0.999)

When budget runs low → freeze non-essential releases, focus on reliability work.

2) Alert on burn rate, not just thresholds

A plain threshold alert fires too late or too early. Burn rate alerts fire when you are consuming your error budget faster than sustainable.

Multi-window, multi-burn-rate alert (Prometheus)

# Fast burn: consuming 14x the budget over 1 hour (fires quickly)
- alert: HighErrorBudgetBurnRate
  expr: |
    (
      job:http_error_rate:rate1h > (14 * 0.001)
      AND
      job:http_error_rate:rate5m > (14 * 0.001)
    )
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "High error budget burn rate on {{ $labels.job }}"
    description: "At this rate, the 30-day budget will be consumed in < 2 hours"

# Slow burn: consuming 6x the budget over 6 hours (fires before noticeable impact)
- alert: ModerateErrorBudgetBurnRate
  expr: |
    (
      job:http_error_rate:rate6h > (6 * 0.001)
      AND
      job:http_error_rate:rate30m > (6 * 0.001)
    )
  for: 15m
  labels:
    severity: warning

3) Incident response lifecycle

Phase 1: Detection

• Alert fires (automated) or user report arrives
• On-call engineer acknowledges within SLA (e.g., 5 minutes for critical)

Phase 2: Declaration

• If impact is significant, declare an incident and open a dedicated channel #incident-YYYY-MM-DD-description
• Assign roles:
  • Incident Commander (IC) — coordinates response, makes decisions
  • Technical Lead — investigates and implements mitigations
  • Comms Lead — updates stakeholders and status page

Phase 3: Investigation

Follow this checklist:

□ What changed recently? (deploys, config changes, infra changes)
□ What is the blast radius? (which services/users are affected)
□ Check the key dashboard: error rate, latency, saturation
□ Check logs for error signatures
□ Check traces for slow/failing operations
□ Check downstream dependencies: databases, external APIs

Phase 4: Mitigation

Prioritize mitigation over root cause—get the system healthy first:

• Roll back the last deploy: kubectl rollout undo deployment/myapp
• Disable a feature flag
• Scale out the affected service
• Re-route traffic to a healthy region

Phase 5: Resolution

• Confirm metrics have returned to normal
• Lift any traffic diversions or feature flags
• Announce resolution to stakeholders

4) Incident runbook template

Every service should have a runbook linked from its alerts.

# Service: Payment API

## SLO
- Success rate: 99.95% over 30 days
- p95 latency: < 300ms

## Key Dashboards
- [Overview dashboard](https://grafana.example.com/d/payment-overview)
- [Error detail](https://grafana.example.com/d/payment-errors)

## Alert: HighErrorRate

**What it means**: More than 0.5% of requests are returning 5xx errors.

**Impact**: Customers cannot complete purchases.

### Investigation steps
1. Check the Error Rate panel: is it increasing or stable?
2. Run log query: `service="payment-api" level="error"` — look for error patterns
3. Check Jaeger for recent failed traces
4. Check the upstream DB: is `payment_db_latency_p95` elevated?
5. Check if any deployment happened in the last 30 minutes

### Mitigation options
- **If caused by a bad deploy**: `kubectl rollout undo deployment/payment-api -n production`
- **If DB overloaded**: enable read replica, reduce connection pool size
- **If third-party payment provider down**: enable offline queue mode via feature flag

## Escalation
- On-call engineer: PagerDuty rotation `payment-team`
- Manager escalation: after 30 minutes unresolved
- Vendor contact: Stripe support if gateway is the issue

5) Blameless post-mortem

Write a post-mortem within 48–72 hours of every major incident.

# Post-Mortem: Payment API Outage — 2024-03-15

## Summary
Payment API experienced a 47-minute elevated error rate (8% failures) caused by
a database connection pool exhaustion after a traffic spike during a marketing campaign.

## Impact
- 3,200 failed checkout attempts
- Estimated revenue impact: ~$48,000
- SLO: consumed 38% of the 30-day error budget

## Timeline
| Time (UTC) | Event |
|------------|-------|
| 14:02 | Deploy of payment-api v2.3.1 to production |
| 14:18 | Error rate alert fires (HighErrorBudgetBurnRate critical) |
| 14:21 | On-call engineer acknowledges, opens incident channel |
| 14:25 | Marketing email campaign send begins (10x normal traffic) |
| 14:29 | Root cause identified: DB connection pool exhausted |
| 14:35 | Mitigation: pool size increased, horizontal scale-out |
| 15:05 | Error rate returns to normal, incident resolved |

## Root cause
Connection pool was sized for normal traffic (max_connections=20).
The marketing campaign triggered a 10x traffic spike that the pool could not handle.
No load testing was performed for marketing campaign scenarios.

## Contributing factors
1. No alert on connection pool utilisation
2. Marketing campaign traffic spike was not communicated to the engineering team
3. Load testing only covered baseline traffic profiles

## What went well
- Alert fired quickly (6 minutes after incident start)
- On-call response was fast (3 minutes to acknowledge)
- Mitigation was effective once root cause was identified

## Action items
| Action | Owner | Due |
|--------|-------|-----|
| Add connection pool utilisation alert | @sre-team | 2024-03-22 |
| Load test for 10x traffic scenarios | @platform-team | 2024-03-29 |
| Create process for comms between marketing and engineering | @eng-manager | 2024-03-22 |
| Implement adaptive connection pooling | @backend-team | 2024-04-05 |

6) Reliability improvement loop

Incident → Post-mortem → Action items → Reliability improvements
    ↑                                              ↓
    └──────────────── Monitor SLO ─────────────────┘

Track action items in your project tracker. Review completion in weekly reliability reviews. Trend your error budget consumption month-over-month—if it improves, your process is working.

Next steps

• Chaos engineering: proactively test failure scenarios with tools like Chaos Monkey
• Toil reduction: automate repetitive ops work to free up time for reliability improvements
• SRE workbook: Google’s free SRE books at sre.google