Lecture 32: Streaming: Kafka Concepts

Content locked

The catalog is public. Reading this content requires approved access.

Overview

Learning Objectives

Sending and Receiving Messages

Procedure Calls

Remote Procedure Calls (RPCs)

Serialization and Deserialization

Synchronous vs. Asynchronous Communication

ETL (Extract Transform Load)

The ETL Problem

Issue 1: Data Freshness

Issue 2: Scaling Engineering Effort

The Unified Log Idea

Kafka Design

Topics

Producers Publish (pub/sub)

Receiving Messages with poll()

What’s in a Message?

Scalability with Partitioning

Why Partition?

Partitions

Append and Delete

Selecting a Partition

Consuming Partitioned Topics

Read Offsets

Partial vs. Total Ordering

Ordering Kafka Messages

Seek to an Offset

Consumer Groups

Why Consumer Groups?

Per-Group Offsets

Partition Assignment

Manual Assignment

Automatic Assignment

Broker Storage: Segment Files

Segment Files

Rollover

Deletion

Log Policy

Conclusion

Source