Keys, Partitioning, and Ordering Guarantees

The default partitioner is nine lines of logic that decide almost everything about how a Kafka topic behaves under load. This lesson makes that logic explicit.

The rule

For a record with a key, the partition is:

partition = murmur2(serialized_key) % number_of_partitions

For a record without a key, the producer uses a sticky assignment — it picks one partition, fills a batch, then moves to another. Neither case involves the broker; the producer decides the partition before the record leaves the process.

You can compute it yourself and predict where a record will land:

def murmur2(data: bytes) -> int:
    """Kafka's partitioner hash — a port of the Java client's implementation."""
    length = len(data)
    seed = 0x9747B28C
    m, r = 0x5BD1E995, 24
    h = (seed ^ length) & 0xFFFFFFFF

    for i in range(0, length - length % 4, 4):
        k = (data[i] | data[i + 1] << 8 | data[i + 2] << 16 | data[i + 3] << 24)
        k = (k * m) & 0xFFFFFFFF
        k ^= k >> r
        k = (k * m) & 0xFFFFFFFF
        h = ((h * m) & 0xFFFFFFFF) ^ k

    rem = length % 4
    if rem >= 3: h ^= data[length - rem + 2] << 16
    if rem >= 2: h ^= data[length - rem + 1] << 8
    if rem >= 1:
        h ^= data[length - rem]
        h = (h * m) & 0xFFFFFFFF

    h ^= h >> 13
    h = (h * m) & 0xFFFFFFFF
    h ^= h >> 15
    return h

def partition_for(key: str, partitions: int) -> int:
    # The Java client masks off the sign bit before taking the modulo.
    return (murmur2(key.encode()) & 0x7FFFFFFF) % partitions

for key in ["alice", "bob", "carol", "dave", "order-42"]:
    print(f"{key:10} -> partition {partition_for(key, 3)}")

$ python partitioner.py
alice      -> partition 2
bob        -> partition 1
carol      -> partition 2
dave       -> partition 0
order-42   -> partition 0

Produce those keys and check — the assignment matches exactly. alice and carol share partition 2, which is normal: with three partitions and five keys, collisions are guaranteed.

Why this is the ordering guarantee

Kafka’s ordering promise is narrow and worth stating precisely: records within one partition are delivered in the order they were written, and nothing else is ordered.

Because a key always maps to one partition, all records sharing a key are in one partition, so they inherit that ordering. This is the whole mechanism. Keying by order_id means an order’s created → paid → shipped sequence can never be observed out of order, even though the topic as a whole has no order at all.

Watch it fail without a key:

from confluent_kafka import Producer

producer = Producer({"bootstrap.servers": "localhost:9092"})
for state in ["created", "paid", "shipped", "delivered"]:
    producer.produce("lifecycle", value=state)   # no key
producer.flush()

bin/kafka-console-consumer.sh --topic lifecycle --from-beginning \
  --property print.partition=true --max-messages 4 \
  --bootstrap-server localhost:9092

Partition:1	paid
Partition:2	shipped
Partition:0	created
Partition:0	delivered
Processed a total of 4 messages

paid arrived before created. A consumer reading this stream would process a payment for an order it has never seen. Nothing is broken — you simply never asked for ordering.

Add the key and re-run:

for state in ["created", "paid", "shipped", "delivered"]:
    producer.produce("lifecycle", key="order-42", value=state)

Partition:0	created
Partition:0	paid
Partition:0	shipped
Partition:0	delivered
Processed a total of 4 messages

One partition, correct order, every time.

The hot-key problem

Keying buys ordering and costs balance. If one key dominates your traffic, its partition does all the work while the others idle:

from confluent_kafka import Producer
from collections import Counter

producer = Producer({"bootstrap.servers": "localhost:9092"})
landed = Counter()

def track(err, msg):
    if not err:
        landed[msg.partition()] += 1

# 90% of traffic from one tenant — a realistic multi-tenant skew
keys = ["tenant-mega"] * 900 + [f"tenant-{i}" for i in range(100)]
for k in keys:
    producer.produce("usage", key=k, value="event", callback=track)
producer.poll(0)
producer.flush()

for p in sorted(landed):
    print(f"partition {p}: {landed[p]:4} records")

$ python hot_key.py
partition 0:   38 records
partition 1:  931 records
partition 2:   31 records

Partition 1 holds 93% of the data. The consumer assigned to it becomes the bottleneck, and adding consumers does not help — no other consumer is allowed to read that partition. This is the standard reason a Kafka pipeline stops scaling.

The fix is a compound key that preserves the ordering you actually need while spreading load. If you only need ordering per (tenant, user) rather than per tenant:

for k in keys:
    user = f"u{hash(k) % 50}"
    producer.produce("usage", key=f"{k}:{user}", value="event", callback=track)

partition 0:  334 records
partition 1:  341 records
partition 2:  325 records

Even distribution, and ordering is still exact within each tenant-user pair.

Custom partitioners

When hashing is not the right rule — routing by region, isolating a tenant onto dedicated partitions — you can supply the partition explicitly:

REGION_PARTITION = {"eu": 0, "us": 1, "apac": 2}

for region, payload in [("eu", "e1"), ("us", "u1"), ("apac", "a1"), ("eu", "e2")]:
    producer.produce(
        "regional",
        key=region,
        value=payload,
        partition=REGION_PARTITION[region],   # bypasses the partitioner entirely
        callback=track,
    )
producer.flush()

Partition:0	eu	e1
Partition:0	eu	e2
Partition:1	us	u1
Partition:2	apac	a1

Explicit partitions are a commitment: the mapping is now in your application code, and changing the partition count means changing that code too. Use it when the routing rule is genuinely a business rule, not to avoid understanding the hash.

Practice

print(partition_for("user-7", 4))

1

printf 'user-7:hello\n' | bin/kafka-console-producer.sh --topic p4 \
  --property parse.key=true --property key.separator=: \
  --bootstrap-server localhost:9092

Partition:1	user-7	hello

for n in (3, 4, 5):
    print(f"{n} partitions -> {partition_for('order-42', n)}")

3 partitions -> 0
4 partitions -> 3
5 partitions -> 3

partition 0:  336 records
partition 1:  332 records
partition 2:  332 records

Next: what “delivered” actually means, and the three delivery guarantees you can choose between.