Lecture 34: Streaming: Kafka Reliability

Content locked

The catalog is public. Reading this content requires approved access.

Overview

Learning Objectives

Kafka Replication

Brokers, Partitions, and Replication Factor

Fetch Requests

In-Sync vs. Lagging Followers

Minimum In-Sync Replicas

Tradeoffs for min.insync.replicas

Fault Tolerance

Leader Election

Failover Behavior

Some Messages May Be Lost on Failover

Committed Messages

What “Committed” Means

Kafka’s Definition

What if There Are No Eligible Leaders?

Working with Committed Data

Example 1: Write Anomaly

Acks From the Programmer’s Perspective

Example 2: Read Anomaly

Fetch Behavior: Consumer vs. Follower

Exactly-Once Semantics

What Are Semantics?

At-Most-Once Semantics

At-Least-Once Semantics: Strong Acks + Retry

Are Duplicates OK? Idempotence

Suppressing Duplicates

Exactly-Once Semantics: Producer Side

Exactly-Once Semantics: Consumer Side

The Auto-Commit Pitfall

Approach 1: Manually Commit Offsets

Approach 2: Externally Save Commits

Conclusion

Source