Do I need ZooKeeper to run Kafka?

No. Kafka 3.3 and later run in KRaft mode, where the brokers manage cluster metadata themselves. ZooKeeper was removed entirely in Kafka 4.0. Every command in this lesson uses KRaft, so there is no ZooKeeper process to start.

What is the difference between Kafka and a message queue like RabbitMQ?

A queue deletes a message once a consumer acknowledges it. Kafka keeps every message in an append-only log for a configured retention period, and each consumer tracks its own position. That means many independent consumers can read the same data, and you can replay history by resetting a position.

Why does my consumer print nothing when I start it?

By default a console consumer starts reading at the end of the log, so it only sees messages produced after it started. Pass --from-beginning to read the topic from offset 0.

Is one broker enough for development?

Yes. A single-broker cluster behaves the same as a multi-broker one for producing and consuming. You only need more brokers to test replication and failover, since a topic cannot have a replication factor higher than the broker count.

Kafka in Ten Minutes: Your First Topic

Kafka is an append-only log you can write to from many places and read from many places at once. Before any theory, it is worth running the whole loop — broker, topic, producer, consumer — because the mental model falls out of watching it work.

Starting a broker

Kafka ships as a tarball with shell scripts in bin/. Since Kafka 3.3 the broker stores its own metadata (KRaft mode), so there is no ZooKeeper to start first. You format a storage directory once, then run the server.

# Download and unpack
curl -sO https://downloads.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz
tar -xzf kafka_2.13-3.9.0.tgz
cd kafka_2.13-3.9.0

# Generate a cluster ID and format the log directory (once, ever)
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/kraft/server.properties

$ bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/kraft/server.properties
Formatting metadata directory /tmp/kraft-combined-logs with metadata.version 3.9-IV0.

Formatting writes a meta.properties file stamped with that cluster ID. It is a one-time step — run it again on the same directory and Kafka refuses, because reformatting would discard the log.

Now start the broker. It runs in the foreground, so give it its own terminal.

bin/kafka-server-start.sh config/kraft/server.properties

[2026-02-14 09:12:41,003] INFO Kafka version: 3.9.0 (org.apache.kafka.common.utils.AppInfoParser)
[2026-02-14 09:12:41,187] INFO [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2026-02-14 09:12:41,191] INFO Kafka startTimeMs: 1771059161189 (org.apache.kafka.common.utils.AppInfoParser)

Transition from STARTING to STARTED is the line that matters. The broker is now listening on localhost:9092.

Creating a topic

A topic is a named log. Create one in a second terminal.

bin/kafka-topics.sh --create \
  --topic orders \
  --partitions 1 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

Created topic orders.

Ask the broker to describe what it just made:

bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Topic: orders	TopicId: 8xRt2mQlSveKp0nZqW3vLA	PartitionCount: 1	ReplicationFactor: 1	Configs:
	Topic: orders	Partition: 0	Leader: 1	Replicas: 1	Isr: 1

Read that second line carefully — it is the whole storage model in one row. The topic has one partition, numbered 0. That partition lives on broker 1, which is its leader. Replicas: 1 and Isr: 1 both list broker 1, because with a single broker there is nowhere else to copy the data.

Producing messages

kafka-console-producer.sh reads lines from stdin and sends each one as a message.

bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

It gives you a > prompt. Type three lines, then press Ctrl+D:

>order-1 espresso
>order-2 cortado
>order-3 flat white
>

There is no confirmation printed per message — silence means the broker acknowledged the write. Kafka only prints here when something fails.

Consuming them back

bin/kafka-console-consumer.sh \
  --topic orders \
  --from-beginning \
  --bootstrap-server localhost:9092

order-1 espresso
order-2 cortado
order-3 flat white

The consumer does not exit. It sits there waiting for more messages, because a Kafka topic is a stream, not a file with an end. Leave it running, go back to the producer terminal, and send a fourth line — it appears in the consumer within milliseconds.

--from-beginning is what made the first three messages appear. Stop the consumer with Ctrl+C, then start it again without that flag:

bin/kafka-console-consumer.sh --topic orders --bootstrap-server localhost:9092

Nothing. The consumer is reading from the end of the log, and nothing new has arrived. This is the single most common “Kafka is broken” moment, and it is just the default auto.offset.reset policy doing its job.

The messages did not disappear

Reading a message does not consume it in the destructive sense. The log still holds all four records, and you can prove it by asking for the current end offset:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic orders

orders:0:4

Read that as topic:partition:offset — partition 0 of orders will assign offset 4 to the next message written, which means offsets 0 through 3 are occupied. Run --from-beginning again and all four records replay, in the same order, as many times as you like. That replayability is the property that separates Kafka from a queue.

Practice

1. Create a topic named payments with 3 partitions and confirm the partition count.

bin/kafka-topics.sh --create --topic payments --partitions 3 \
  --replication-factor 1 --bootstrap-server localhost:9092
bin/kafka-topics.sh --describe --topic payments --bootstrap-server localhost:9092

Created topic payments.
Topic: payments	TopicId: qP9vK2mXTdyLs4rNzB8wEg	PartitionCount: 3	ReplicationFactor: 1	Configs:
	Topic: payments	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: payments	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: payments	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

All three partitions land on broker 1 — with one broker there is no other option.

2. Produce five messages to payments, then check the end offset of each partition. What do you notice?

bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic payments

payments:0:2
payments:1:1
payments:2:2

The five messages were spread across the three partitions rather than all landing in partition 0. Console-producer messages have no key, so Kafka distributes them. This is also why consuming a multi-partition topic does not give you global ordering — covered in Topics, Partitions, and Offsets.

3. Try to create a topic with --replication-factor 3 on your single broker. What happens, and why?

Error while executing topic command : Unable to replicate the partition 3 time(s): The target replication factor of 3 cannot be reached because only 1 broker(s) are registered.

A replica is a full copy of a partition on a different broker. Kafka will not place two replicas of the same partition on one broker, because that would provide no protection against that broker failing — so the replication factor can never exceed the broker count.

4. Delete the orders topic, recreate it, and check the end offset. Where did the offsets go?

bin/kafka-topics.sh --delete --topic orders --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic orders --partitions 1 \
  --replication-factor 1 --bootstrap-server localhost:9092
bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic orders

orders:0:0

Offsets are per-partition positions in a specific log, not global IDs. Deleting the topic deletes the log, so the recreated topic starts from 0 again. Offsets are only meaningful within the lifetime of a partition.

Next, the structure underneath all of this: how partitions and offsets actually decide what order your data arrives in.