What is the ISR and why does it shrink?

The in-sync replica set is the replicas that have caught up with the leader within replica.lag.time.max.ms (default 30s). A replica drops out when it falls behind — usually from GC pauses, slow disks, or network saturation — and rejoins once it catches up. A shrinking ISR is the earliest warning that a broker is struggling.

Why does acks=all not guarantee durability on its own?

acks=all waits for all in-sync replicas, but if the ISR has shrunk to just the leader then 'all' means one. Pair it with min.insync.replicas=2 so the broker rejects writes rather than accepting them with no redundancy.

When should I use compaction instead of time-based retention?

Use compaction when the topic represents current state rather than a series of events — user profiles, feature flags, the latest price per instrument. Compaction keeps at least the newest record per key forever, so a consumer can rebuild full state by replaying from offset 0.

Does compaction remove duplicates immediately?

No. The active segment is never compacted, and the log cleaner runs on a background schedule against closed segments. A compacted topic will always contain some superseded records; compaction bounds growth rather than guaranteeing uniqueness at any instant.

Replication, ISR, Retention, and Compaction

Everything so far assumed one broker. Production has several, and the interesting behaviour is what happens when one goes away.

A three-broker cluster

Start three brokers by copying the config and changing three values per node:

for id in 1 2 3; do
  sed -e "s/^node.id=.*/node.id=$id/" \
      -e "s#^log.dirs=.*#log.dirs=/tmp/kraft-$id#" \
      -e "s/^listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/:909$((1+id))/" \
      -e "s/^advertised.listeners=.*/advertised.listeners=PLAINTEXT:\/\/localhost:909$((1+id))/" \
      config/kraft/server.properties > config/kraft/server-$id.properties
done

With all three running, create a replicated topic:

bin/kafka-topics.sh --create --topic payments \
  --partitions 3 --replication-factor 3 \
  --bootstrap-server localhost:9092

bin/kafka-topics.sh --describe --topic payments --bootstrap-server localhost:9092

Created topic payments.
Topic: payments	PartitionCount: 3	ReplicationFactor: 3	Configs: min.insync.replicas=1
	Topic: payments	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: payments	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: payments	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

Each partition has one leader and two followers. All reads and writes go through the leader; followers only replicate. Kafka spread the leadership so each broker leads one partition — that is how the cluster balances load.

Isr: 1,2,3 means all three replicas are caught up.

Killing the leader

Stop broker 2, which leads partition 1:

kill $(jps | grep Kafka | awk '$0 ~ /9093/ {print $1}')
bin/kafka-topics.sh --describe --topic payments --bootstrap-server localhost:9092

	Topic: payments	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,3
	Topic: payments	Partition: 1	Leader: 3	Replicas: 2,3,1	Isr: 3,1
	Topic: payments	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1

Partition 1’s leader moved from broker 2 to broker 3, automatically, in under a second. Broker 2 has dropped out of every ISR. Producers and consumers reconnect to the new leader without any application change — the client library refreshes metadata on the NOT_LEADER_OR_FOLLOWER error and retries.

Restart broker 2 and it catches up:

	Topic: payments	Partition: 1	Leader: 3	Replicas: 2,3,1	Isr: 3,1,2

Back in the ISR, but not leader again. Kafka does not move leadership back on its own unless auto.leader.rebalance.enable is set, because leader elections are disruptive. Over a series of failures leadership drifts and the cluster becomes unbalanced — run kafka-leader-election.sh --election-type preferred to restore it.

Why acks=all needs min.insync.replicas

With min.insync.replicas=1, a write is acknowledged even when only the leader survives:

bin/kafka-configs.sh --alter --entity-type topics --entity-name payments \
  --add-config min.insync.replicas=2 --bootstrap-server localhost:9092

Completed updating config for topic payments.

Now stop two brokers, leaving one, and produce with acks=all:

$ python produce.py
FAILED: KafkaError{code=NOT_ENOUGH_REPLICAS,val=19,
  str="Messages are rejected since there are fewer in-sync replicas than required."}

The broker refuses the write. That is the correct behaviour: accepting it would mean a record with exactly one copy, which the surviving broker’s disk failure would erase. The producer’s job is now to back off and retry, not to push data into a cluster that cannot protect it.

The standard durable configuration is replication-factor=3, min.insync.replicas=2, acks=all. It tolerates one broker failure with no data loss and no write interruption.

Retention: deleting by time and size

By default Kafka deletes segments older than a week:

bin/kafka-configs.sh --describe --entity-type topics --entity-name payments \
  --bootstrap-server localhost:9092

Dynamic configs for topic payments are:
  min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:min.insync.replicas=2}

Set a short retention to watch it work:

bin/kafka-configs.sh --alter --entity-type topics --entity-name payments \
  --add-config retention.ms=60000,segment.ms=10000 \
  --bootstrap-server localhost:9092

Produce some records, wait, and check the start offset:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic payments --time -2   # earliest
bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic payments --time -1   # latest

payments:0:0
payments:0:150

After 90 seconds:

payments:0:120
payments:0:150

The earliest available offset moved from 0 to 120 — those records are gone from disk. A consumer that was sitting at offset 50 now gets:

OFFSET_OUT_OF_RANGE: Fetch position 50 is out of range for partition payments-0

and resets according to auto.offset.reset, silently skipping to 120 with earliest or to 150 with latest. This is data loss caused by consumer lag exceeding retention, and it is why lag alerting matters more than it first appears.

Retention is per-topic and can be size-based instead:

bin/kafka-configs.sh --alter --entity-type topics --entity-name payments \
  --add-config retention.bytes=1073741824 --bootstrap-server localhost:9092

Both limits apply — whichever triggers first.

Compaction: keeping the latest value per key

For a topic that represents state rather than events, deleting by age is wrong. You want the newest record per key kept indefinitely.

bin/kafka-topics.sh --create --topic user-profiles \
  --partitions 1 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.01 \
  --config segment.ms=5000 \
  --bootstrap-server localhost:9092

Created topic user-profiles.

Write several versions of the same keys:

printf 'alice:{"plan":"free"}\nbob:{"plan":"free"}\nalice:{"plan":"pro"}\nbob:{"plan":"pro"}\nalice:{"plan":"enterprise"}\n' | \
  bin/kafka-console-producer.sh --topic user-profiles \
    --property parse.key=true --property key.separator=: \
    --bootstrap-server localhost:9092

Read immediately:

alice	{"plan":"free"}
bob	{"plan":"free"}
alice	{"plan":"pro"}
bob	{"plan":"pro"}
alice	{"plan":"enterprise"}

All five — nothing has been compacted yet. Produce a few more records to roll the segment, wait for the cleaner, then read again:

bob	{"plan":"pro"}
alice	{"plan":"enterprise"}

Superseded records are gone; the latest value per key remains. A new consumer replaying from offset 0 reconstructs the current state of every user without reading the full history. This is the mechanism behind Kafka Streams state stores and Connect’s offset topics.

Deleting a key with a tombstone

A null value marks a key for deletion:

printf 'bob:\n' | bin/kafka-console-producer.sh --topic user-profiles \
  --property parse.key=true --property key.separator=: \
  --property null.marker=  --bootstrap-server localhost:9092

alice	{"plan":"enterprise"}
bob	null

The tombstone is retained for delete.retention.ms (default 24 hours) so every consumer has a chance to observe the deletion, then it is removed too. Compacting away a tombstone too early leaves consumers that were offline with a stale value they will never be told to discard — which is exactly what that 24-hour default protects against.

Practice

1. Create a topic with replication-factor 3, stop one broker, and produce with acks=all and min.insync.replicas=2. Does it succeed?

Yes. Two replicas remain in the ISR, which satisfies the minimum. Stop a second broker and it fails with NOT_ENOUGH_REPLICAS. This is the intended envelope: RF=3 with min.insync.replicas=2 survives exactly one failure while still accepting writes.

2. Set retention.ms=1000 on a topic and try to consume from the beginning.

Processed a total of 0 messages

Records are deleted almost as fast as they are written. retention.ms sets the minimum age before a closed segment becomes eligible, so the active segment survives briefly — but a consumer restarting into this topic sees essentially nothing. Very short retention is only sensible on topics whose consumers are guaranteed to be live.

3. On a compacted topic, does the offset of a surviving record change after compaction?

No. Compaction removes records but never renumbers the ones that remain, so offsets stay stable and become non-contiguous:

Offset:1	bob	{"plan":"pro"}
Offset:4	alice	{"plan":"enterprise"}

Gaps in a compacted log are normal. Code that assumes offset + 1 is the next record will break here — always use the offset the consumer hands you.

4. Your consumer group has 2 million lag and retention is 24 hours. What is the risk?

If the group cannot catch up within 24 hours, the broker deletes records it has not read yet, and on the next fetch it gets OFFSET_OUT_OF_RANGE and jumps forward — losing data silently, with no error in the consumer. The mitigations are to alert on lag as a fraction of retention rather than as a raw number, raise retention during an incident (it takes effect immediately), and scale consumers up to the partition count.

Next: tuning throughput and latency once the correctness settings are right.