Deck 10: Kafka

Deck 10: Kafka

Q1. What is the problem with “auto commit” for Kafka consumers?

Q2. If we have Kafka consumers (a) processing messages from a Kafka topic and (b) writing output based on that processing, how can we guarantee each message is considered exactly once in the output, even if the consumer crashes and restarts?

Q3. When is a Kafka message considered “committed”?

Q4. In Kafka, both consumers and follower replicas fetch data from the leader replica.  But how do the fetches differ?

Q5. If min.insync.replicas is X and the number of currently in-sync replicas is Y, why doesn’t Kafka consider a message committed as soon as it is written to at least X replicas?  (assume X<Y in this question)

Q6. In order to be considered “in sync”, does a follower replica in Kafka need to have ALL the messages the leader has seen?

Q7. How do “elections” in distributed systems usually differ from a regular democratic election humans might participate in?

Q8. What is an example of a “write anomoly” in Kafka?

Q9. What is an example of a “read anomoly” in Kafka?

Q10. Producers can be created like this: KafkaProducer(…, acks=????)

Q11. How might we end up with “at-least-once” semantics?

Q12. How might you end up with “at-most-once” semantics?

Q13. What is the difference between log “rollover” and “deletion” in Kafka?  What does “retention” mean?

Q14. If your want to connect to a Kafka broker “localhost:9092”, how would you do it in these cases?

Q15. How can you create a Kafka topic “T” with P partitions and R replicas?

Q16. How can a Kafka producer send the string “hello” to topic “greetings”?

Q17. How does a Kafka consumer check for a new batch of messages, with timeout of 1 second?

Q18. How can you see the topic partitions a Kafka consumer is responsible for?

Q19. If you want a Kafka consumer to go back the start of each topic partition, how can you do that?

Q20. Assignment of topic partitions to a Kafka consumer can happen automatically or manually.  What does the code for each look like?

Q21. If tp is a topic partition, how can you check the next offset a Kafka consumer will read at for that partition?

Q22. Kafka consumers typically read messages at ever increasing offsets in each of the tracked topic partitions, but if consumers want to move to a different offset, they can call this:

Q23. Why might you want to have different Kafka consumer groups?

Q24. Why might you want multiple Kafka consumers in the same consumer group?

Q25. Where is topics data stored in Kafka?

Q26. Kafka topics consist of one or more _____, each of which contains a series of ________

Q27. When a producer sends a message, what are 3 things it typically specifies?  And of the 3 things, what is optional?

Q28. If a message doesn’t have a key, how does a Kafka consumer decide what topic partition it should go in?

Q29. If a Kafka producer specifies a key for a message, how does it determine what partition the message should go to?

Q30. How does a Kafka consumer request a batch of messages from the brokers, with a timeout of N milliseconds?

Q31. What does a Kafka batch contain?

Q32. If Kafka message A was produced before message B, when can we guarantee that consumers will see A before B?

Q33. What are two communication systems we’ve learned, for synchronous and asynchronous communication, respectively?

Q34. What are the three kinds of processes that work together in Kafka?

Q35. What are some advantages of using a streaming platform like Kafka for ETL jobs, rather than running ETL jobs on a schedule (say with cron) to move data from system A to system B?