Apache Kafka Fundamentals

Topics, partitions and offsets

A topic is a named stream of events, like a folder for one kind of event. Examples: orders, payments or page-views.

Each topic is split into partitions. A partition is one ordered, append-only log. Each event in it gets a number that keeps growing, called its offset (its position in the log).

Why partitions?

• Scale. Partitions are spread across many servers, so a topic can hold far more data than one machine can.
• Parallelism. Different consumers can read different partitions at the same time. Parallelism means doing many things at once.
• Ordering. Kafka guarantees order within a partition, not across the whole topic.

Keys decide the partition

Every event can have a key, such as an order ID. A hash is a calculation that turns a key into a number. Kafka hashes the key to choose a partition, so all events with the same key go to the same partition, and stay in order:

key = order_id "5521"  → hash → Partition 1:  created → paid → shipped   ✅ in orderkey = order_id "7730"  → hash → Partition 0:  created → cancelled         ✅ in order

Choosing the key is like choosing a shard key (lesson 25):

• User ID → all of one user's events in order.
• Order ID → each order's lifecycle in order.
• No key → events are spread evenly, with no ordering guarantee.
• A hot key (one very busy key, such as one huge customer) → one overloaded partition.

Brokers and replication

Kafka runs as a cluster (a group of servers that work together). Each server is called a broker. Each partition is replicated (copied) to several brokers, commonly 3:

Partition 0:  Leader on Broker 1,  followers on Broker 2 and 3Partition 1:  Leader on Broker 2,  followers on Broker 3 and 1Partition 2:  Leader on Broker 3,  followers on Broker 1 and 2

• The leader handles reads and writes for that partition. Followers copy it (single-leader replication, lesson 24).
• The ISR (In-Sync Replicas) are the followers that are caught up with the leader.
• If a leader's broker dies, an in-sync follower becomes the new leader.

Durability settings decide how safe writes are:

• acks=0: the producer does not wait at all. (acks is the setting for how many brokers must confirm a write.) Fastest, and messages can be lost.
• acks=1: wait for the leader only. The message is lost if the leader dies before followers copy it.
• acks=all: wait for all in-sync replicas. Combined with min.insync.replicas=2 (with a replication factor of 3, meaning 3 copies), a write is confirmed only when at least two brokers have it. This is the setting for important data.

Metadata: from ZooKeeper to KRaft

Kafka needs to track which brokers are alive, who leads each partition, and topic configurations. For years this used Apache ZooKeeper, a separate system that stores shared settings and status. Kafka then built its own system based on Raft (a way for servers to agree), called KRaft. Kafka 4.0 removed ZooKeeper completely. The result is fewer parts to run and faster recovery.

Producers

Producers send events to topics. Features that matter:

• Batching: collect many small messages and send them together (settings like linger.ms and batch.size), for much higher throughput.
• Compression: compress batches (for example with lz4 or zstd) to save network and disk.
• Idempotent producer: idempotent means repeating it has the same effect as doing it once. Kafka removes duplicates caused by producer retries, so a network retry does not write the same event twice. It is on by default in modern Kafka (since version 3.0).
• Transactions: write to several partitions atomically (all succeed or none do; lesson 37).

Consumers and consumer groups

Consumers read events and track their offset, meaning "I have processed everything up to here".

Consumers work in consumer groups (lesson 35). A consumer group is a team of consumers that share the work of one topic:

• Each partition is read by exactly one consumer in the group at a time.
• Different groups read the same topic independently, each with its own offsets.

Topic "orders" (4 partitions)Group "billing" (2 consumers):   consumer A ← P0, P1   consumer B ← P2, P3Group "analytics" (4 consumers):   c1 ← P0   c2 ← P1   c3 ← P2   c4 ← P3Group "search" (5 consumers):   4 get one partition each… the 5th is IDLE (no partition left)

Important rule: the number of partitions is the maximum parallelism for a consumer group. More consumers than partitions leaves some idle. Choose partition counts with future growth in mind. You can add partitions later, but that changes which partition each key maps to.

Rebalancing. When a consumer joins, leaves or crashes, Kafka shares the partitions out again among the group. During a rebalance, reading can pause for a short time. Modern Kafka offers a smoother "cooperative" rebalancing.

Committing offsets. Consumers periodically commit (save) their offset back to Kafka (stored in an internal topic). When you commit decides your delivery guarantee:

• commit before processing → possible loss if you crash,
• commit after processing → possible duplicates if you crash.

(Lesson 37 covers this in depth.)

Consumer lag is how far behind the newest event a consumer is. It is the most important Kafka metric to watch. Rising lag means consumers cannot keep up: scale them, fix slow processing, or check for errors.

Retention and log compaction

Retention means how long data is kept. Kafka keeps events after they are read. You choose how long:

• Time-based retention: keep 7 days (a common default), 30 days, or forever.
• Size-based retention: keep up to X GB per partition.
• Log compaction: keep only the latest event for each key, and discard older ones for that key.

Compaction turns a topic into a replayable table of current state. It is great for things like "current profile of every user" or "current price of every product". A new service can read the whole compacted topic to build its own copy of that data.

Why Kafka is so fast

• Sequential disk I/O. Sequential means in order, one after another. Appending to a log and reading it in order are both very fast on disks (lesson 10).
• The operating system's page cache. This is memory that the operating system uses to keep recent file data. Recent data is usually served from memory without extra work from Kafka.
• Zero-copy transfer. Data goes from the disk cache straight to the network connection without being copied through the application.
• Batching and compression everywhere.
• Partitioning spreads load across many brokers and disks.

The Kafka ecosystem

• Kafka Connect: a tool with ready-made connectors to move data into Kafka (from databases via CDC (change data capture, which reads the database's change log) tools like Debezium, from files and APIs) and out to other systems (Elasticsearch, S3, data warehouses).
• Schema Registry: stores Avro, Protobuf or JSON schemas for events and enforces compatible changes, much like the Protobuf rules in lesson 31. A schema is a description of the fields in an event.
• Kafka Streams / ksqlDB: process streams inside your app (filter, join, aggregate) while keeping state (remembered data) locally.
• Apache Flink: a powerful, separate stream-processing engine commonly paired with Kafka.
• Managed Kafka: Confluent Cloud, Amazon MSK, Aiven, and Kafka-compatible alternatives like Redpanda.

When Kafka is a good fit

• Event streaming between many services (lesson 39).
• High-volume data pipelines: clickstreams, logs, metrics, IoT.
• Change Data Capture: streaming every database change to search, caches and warehouses.
• Event sourcing and audit logs: keeping the full history of changes. (Event sourcing means storing every change as an event instead of only the latest state.)
• Stream processing: real-time analytics, fraud detection, alerting.