ori z blog
rss

Career Guide 18 min read

Part 10: Distributed Systems & Event Streaming — Apache Kafka

Master distributed commit logs, event-driven architectures, Kafka partitions scaling, KRaft consensus quorums, consumer rebalancing, and DLQ error handlers. Complete with 30 curated resources.

By

Part 10: Distributed Systems & Event Streaming — Apache Kafka

← Back to Master Index

1. Core Concept Refresher: The Distributed Append-Only Commit Log

In monolithic backend architectures, application components communicate via direct, synchronous in-memory function calls. When these systems are decomposed into distributed services, developers often attempt to use synchronous HTTP/REST or gRPC calls for inter-service communication.

  • The Coupling Trap: If Service A calls Service B, and Service B calls Service C, the network latency compounds. More dangerously, if Service C crashes, the failure cascades instantly upstream, taking down the entire system.
  • The Escape Hatch: To build highly resilient, loosely coupled platforms at massive scale, software architects transition to Event-Driven Architecture (EDA), with Apache Kafka serving as the core distributed message log.

To succeed in systems engineering roles, you must look past basic publish-subscribe abstractions and master Kafka's physical storage layouts and clustering quorums.

What is Apache Kafka?

Kafka is not a simple message queue like RabbitMQ or ActiveMQ. It is a Distributed Append-Only Commit Log designed to process massive streams of real-time data.

  • Persistent Storage: Unlike traditional message brokers that delete messages as soon as they are acknowledged by consumers, Kafka writes messages to disk sequentially, persisting them based on configurable retention policies (e.g., store for 7 days, or keep forever).
  • Dumb Broker, Smart Client: Traditional brokers track which messages have been read by which consumers, introducing database overhead. Kafka brokers do not maintain this state. Instead, messages are read by consumers who track their own progress using a numeric pointer known as an Offset. This allows multiple consumer groups to read the same data streams at their own pace without interference.

Topics, Partitions, and Consumer Groups

  • Topics: A logical stream of messages, analogous to a database table.
  • Partitions: Topics are physically divided into partitions. A partition is an ordered, immutable sequence of records that is continually appended to. Partitions are the unit of scalability in Kafka. While a single partition can only sit on a single broker, a topic with 12 partitions can be distributed across 12 brokers, allowing writes and reads to execute in parallel.
  • Message Keys & Ordering: Kafka guarantees strict message ordering only within a single partition. If a producer writes messages without a key, the broker distributes them round-robin across partitions, losing overall order. If the producer includes a message key (e.g. user_id), Kafka hashes the key to route all messages with that key to the same partition, guaranteeing ordered processing for that specific user.
  • Consumer Groups: A group of consumers that cooperate to read a topic. Each partition of a topic is assigned to exactly one consumer instance within a consumer group. If you have 4 partitions and 4 consumers in a group, each consumer reads 1 partition. If you add a 5th consumer, it sits idle. Therefore, you cannot scale consumers beyond the number of partitions.
graph TD
    subgraph Topic A (4 Partitions)
        p0["Partition 0 (Append-Only Log)"]
        p1["Partition 1 (Append-Only Log)"]
        p2["Partition 2 (Append-Only Log)"]
        p3["Partition 3 (Append-Only Log)"]
    end
    subgraph Consumer Group A
        c1["Consumer 1"]
        c2["Consumer 2"]
        c3["Consumer 3"]
        c4["Consumer 4"]
    end
    p0 --> c1
    p1 --> c2
    p2 --> c3
    p3 --> c4

Zookeeperless Kafka: KRaft consensus mode

Historically, Kafka relied on Apache ZooKeeper to coordinate brokers, elect controllers, manage configurations, and track topic metadata.

  • The ZooKeeper Bottleneck: ZooKeeper replicated metadata in a separate cluster, creating synchronization delays. Upgrading metadata was slow, and ZooKeeper could not support clusters scaling past 200,000 partitions.
  • The KRaft Solution: Kafka 3.0+ introduced KRaft (Kafka Raft Metadata Mode), which fully deprecates ZooKeeper in Kafka 4.0+. In KRaft mode, metadata management is handled directly inside the Kafka cluster using a built-in consensus protocol based on Raft. Select Kafka brokers act as controllers, maintaining an active, in-memory metadata log that synchronizes instantly, allowing fast controller elections and scaling to millions of partitions.

2. Kafka Master Resource Directory (30 Curated Resources)

Distributed messaging requires deep theoretical understanding of consensus protocols, network layouts, and stream semantics. Below are the essential resources.

Sub-Topic A: Pub-Sub vs. Message Queue Models

1. Designing Data-Intensive Applications

  • Direct URL: https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/
  • Search Identification: Search O'Reilly Media for: "Designing Data-Intensive Applications" (Author: Martin Kleppmann)
  • Resource Type: Book
  • Access / Price: Paid (Included in TCS O'Reilly Enterprise benefit)
  • Status: Required (Non-Negotiable)
  • Description: Landmark systems textbook. Chapter 11 explains the differences between commit logs, event streams, AMQP brokers, and database replication boundaries.
  • Mutual Exclusivity Mapping: If you read this book, you can skip Distributed Messaging Patterns (LinkedIn) as Martin covers messaging models with deeper theoretical detail.

2. Apache Kafka 4.0 Masterclass - Complete Beginners Guide

  • Direct URL: https://www.udemy.com/course/apache-kafka/
  • Search Identification: Search Udemy for: "Apache Kafka 4.0 Masterclass" (Instructor: Stephane Maarek)
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Udemy Business)
  • Status: Required (Non-Negotiable)
  • Description: The undisputed premier video masterclass covering brokers, topics, partitions, CLI commands, and consumer group rebalances.
  • Mutual Exclusivity Mapping: Gold standard practical course; no direct equivalent.

3. Designing Event-Driven Systems

  • Direct URL: https://www.confluent.io/designing-event-driven-systems/
  • Search Identification: Search Google/Web for: "Designing Event-Driven Systems book by Ben Stopford Confluent"
  • Resource Type: Book
  • Access / Price: 100% Free
  • Status: Required
  • Description: Explains how to decompose monolithic databases into event streams and CQRS architectures.
  • Mutual Exclusivity Mapping: Essential architecture reference guide.

4. Distributed Messaging Patterns (LinkedIn Learning)

  • Direct URL: https://www.linkedin.com/learning/distributed-messaging-patterns
  • Search Identification: Search LinkedIn Learning for: "Distributed Messaging Patterns"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Enterprise Account)
  • Status: Alternative to: Designing Data-Intensive Applications.
  • Description: Introduces point-to-point queues, publish-subscribe brokers, and routing keys.
  • Mutual Exclusivity Mapping: Shorter video alternative.

5. RabbitMQ vs Apache Kafka (Coursera)

  • Direct URL: https://www.coursera.org/learn/message-brokers
  • Search Identification: Search Coursera for: "Message Brokers in Distributed Systems"
  • Resource Type: Video Course
  • Access / Price: Free Audit Tier Available
  • Status: Optional
  • Description: Compares RabbitMQ's exchange routing models with Kafka's commit log storage models.
  • Mutual Exclusivity Mapping: Optional booster.

Sub-Topic B: Kafka Topics, Partitions, and Consumer Groups

6. Kafka: The Definitive Guide (2nd Edition)

  • Direct URL: https://www.oreilly.com/library/view/kafka-the-definitive/9781492043072/
  • Search Identification: Search O'Reilly Media for: "Kafka: The Definitive Guide" (Authors: Gwen Shapira, Todd Palino, Rajini Sivaram)
  • Resource Type: Book
  • Access / Price: Paid (Included in TCS O'Reilly Enterprise benefit)
  • Status: Required (Non-Negotiable)
  • Description: The ultimate standard reference manual for producer batching, consumer group offsets commit policies, and partition rebalancing algorithms.
  • Mutual Exclusivity Mapping: If you read this book, you can skip Scaling Apache Kafka (LinkedIn) as Gwen Shapira details client configuration parameters with deeper technical rigor.

7. Apache Kafka for Event-Driven Microservices

  • Direct URL: https://www.udemy.com/course/apache-kafka-for-microservices/
  • Search Identification: Search Udemy for: "Apache Kafka for Event-Driven Spring Boot Microservices" (Instructor: Nam Ha Minh)
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Udemy Business)
  • Status: Alternative to: Kafka: The Definitive Guide (2nd Edition).
  • Description: Video walkthrough configuring partition assignments, consumers groups, and heartbeats.
  • Mutual Exclusivity Mapping: Choose this if you prefer a framework-driven video guide.

8. Kafka Consumer & Producer Internals

  • Direct URL: https://developer.confluent.io/courses/architecture/get-started/
  • Search Identification: Search Web for: "Confluent Developer Kafka internals and client architecture"
  • Resource Type: Written Reference & Interactive Tutorials
  • Access / Price: 100% Free
  • Status: Required
  • Description: Confluent's official visual learning path detailing socket buffers, metadata checks, and partition allocations.
  • Mutual Exclusivity Mapping: Standard reference index.

9. Scaling Apache Kafka: Partitions & Consumer Groups

  • Direct URL: https://www.linkedin.com/learning/scaling-apache-kafka-partitions-and-consumer-groups
  • Search Identification: Search LinkedIn Learning for: "Scaling Apache Kafka"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Enterprise Account)
  • Status: Required
  • Description: Visual walks explaining how partition counts determine consumer scalability bounds and overall cluster capacity.
  • Mutual Exclusivity Mapping: Essential scaling guide.

10. Kafka Interactive Partition CLI Sandbox

  • Direct URL: https://github.com/conduktor/kafka-stack-docker-compose
  • Search Identification: Search GitHub for: "conduktor kafka-stack-docker-compose"
  • Resource Type: Interactive Code Template / Playground
  • Access / Price: 100% Free
  • Status: Optional
  • Description: Multi-broker local Docker script to practice creating topics and scaling partitions via CLI commands.
  • Mutual Exclusivity Mapping: Standard local practice playground.

Sub-Topic C: Replication, Consensus, and Producer ACKS

11. Kafka Core Internals and Replication Protocols

  • Direct URL: https://www.linkedin.com/learning/kafka-core-internals-and-replication-protocols
  • Search Identification: Search LinkedIn Learning for: "Kafka Core Internals"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Enterprise Account)
  • Status: Required (Non-Negotiable)
  • Description: Exceptional walkthrough of replication pathways, Leader/Follower sync logs, and the role of In-Sync Replicas (ISR).
  • Mutual Exclusivity Mapping: Essential replication architecture guide.

12. Complete Guide to Kafka Security and Clustering (Udemy)

  • Direct URL: https://www.udemy.com/course/kafka-security/
  • Search Identification: Search Udemy for: "Apache Kafka Security" (Instructor: Stephane Maarek)
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Udemy Business)
  • Status: Required
  • Description: Video series configuring SSL encryption, SASL authentication, and access control lists (ACLs) between replication nodes.
  • Mutual Exclusivity Mapping: Essential security cluster blueprint.

13. Kafka Replication Protocols Deep Dive

  • Direct URL: https://www.oreilly.com/library/view/kafka-the-definitive/9781492043072/
  • Search Identification: Search O'Reilly Media for: "Kafka: The Definitive Guide" (Chapter 5: Physical Storage Internals)
  • Resource Type: Book Chapter / Spec
  • Access / Price: Paid (Included in TCS O'Reilly Enterprise benefit)
  • Status: Required
  • Description: Low-level specification explaining watermark offsets, leader epochs, and partition state machines.
  • Mutual Exclusivity Mapping: Standard reference specs.

14. Event Streams Replication & Consensus (Coursera)

  • Direct URL: https://www.coursera.org/learn/cloud-computing
  • Search Identification: Search Coursera for: "Cloud Computing Concepts, Part 2" (University of Illinois)
  • Resource Type: Video Course
  • Access / Price: Free Audit Tier Available
  • Status: Alternative to: Kafka Core Internals and Replication Protocols.
  • Description: Academic course explaining distributed consensus, Paxos, Raft, and active replication.
  • Mutual Exclusivity Mapping: Shorter academic alternative.

15. Jepsen Analysis: Apache Kafka Consensus Testing

  • Direct URL: https://jepsen.io/analyses/kafka-0.9.0.0
  • Search Identification: Search Web for: "Jepsen analysis Apache Kafka partition tolerance"
  • Resource Type: Written Analysis / Research Paper
  • Access / Price: 100% Free
  • Status: Optional
  • Description: Technical case study showing how network partitions cause split-brain scenarios and data loss under improper ACK setups.
  • Mutual Exclusivity Mapping: Advanced failure diagnostics reference.

Sub-Topic D: Partition Rebalancing & Consumer Heartbeats

16. Kafka Consumer Group Rebalancing protocols

  • Direct URL: https://www.oreilly.com/library/view/kafka-the-definitive/9781492043072/
  • Search Identification: Search O'Reilly Media for: "Kafka: The Definitive Guide" (Chapter 4: Consumers and Consumer Groups)
  • Resource Type: Book Chapter / Reference
  • Access / Price: Paid (Included in TCS O'Reilly Enterprise benefit)
  • Status: Required (Non-Negotiable)
  • Description: Details the role of the Group Coordinator, heartbeats (heartbeat.interval.ms), and rebalance strategies (Range, RoundRobin, Cooperative Sticky).
  • Mutual Exclusivity Mapping: Standard reference specs.

17. Advanced Kafka Consumer Architecture (Udemy)

  • Direct URL: https://www.udemy.com/course/advanced-kafka-consumers/
  • Search Identification: Search Udemy for: "Advanced Kafka Consumers and Performance Optimization"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Udemy Business)
  • Status: Required
  • Description: Video course configuring multi-threaded consumer handlers, tuning poll loops (max.poll.interval.ms), and managing commits.
  • Mutual Exclusivity Mapping: Essential consumer scaling companion.

18. Kafka Rebalance Protocols Deep Dive

  • Direct URL: https://www.linkedin.com/learning/kafka-rebalance-protocols
  • Search Identification: Search LinkedIn Learning for: "Kafka Rebalance Protocols"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Enterprise Account)
  • Status: Alternative to: Advanced Kafka Consumer Architecture.
  • Description: Explains incremental cooperative rebalancing algorithms and how they eliminate cluster-wide "stop-the-world" pauses.
  • Mutual Exclusivity Mapping: Shorter video alternative.

19. Confluent Blog: Cooperative Sticky Rebalance Explainer

  • Direct URL: https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-connect-a-deep-dive/
  • Search Identification: Search Google/Web for: "Confluent Deep Dive into the Cooperative Rebalancing Protocol"
  • Resource Type: Written Case Study / Blog
  • Access / Price: 100% Free
  • Status: Required
  • Description: Exceptional analysis explaining why static membership and sticky assignors reduce rebalance overhead.
  • Mutual Exclusivity Mapping: Essential operational reference.

20. Kafka Consumer Group Lag Monitor Sandbox (Lagman)

  • Direct URL: https://github.com/linkedin/Burrow
  • Search Identification: Search GitHub for: "linkedin burrow kafka consumer lag monitoring"
  • Resource Type: Code Library / Interactive Tool
  • Access / Price: 100% Free
  • Status: Optional
  • Description: LinkedIn's open-source tool Burrow to monitor consumer lags without tracking active Zookeeper dependencies.
  • Mutual Exclusivity Mapping: Optional observability dashboard.

Sub-Topic E: Dead Letter Queues (DLQ) & Error Handling

21. Error Handling Patterns in Apache Kafka

  • Direct URL: https://www.linkedin.com/learning/error-handling-patterns-in-apache-kafka
  • Search Identification: Search LinkedIn Learning for: "Error Handling Patterns in Apache Kafka"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Enterprise Account)
  • Status: Required (Non-Negotiable)
  • Description: Details the implementation of Dead Letter Queues (DLQ), retry topics, and blocking vs. non-blocking retry mechanisms.
  • Mutual Exclusivity Mapping: Essential event-driven resilience guide.

22. Designing Fault-Tolerant Event-Driven Architectures (Udemy)

  • Direct URL: https://www.udemy.com/course/fault-tolerant-event-driven/
  • Search Identification: Search Udemy for: "Designing Fault-Tolerant Event-Driven Architectures"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Udemy Business)
  • Status: Required
  • Description: Setting up retry loops, idempotency parameters, and transactional outbox patterns to achieve exactly-once processing (EOS).
  • Mutual Exclusivity Mapping: High-end architectural safety manual.

23. Confluent: Kafka Error Handling Best Practices

  • Direct URL: https://developer.confluent.io/patterns/error-handling/
  • Search Identification: Search Web for: "Confluent Developer Kafka error handling patterns dead letter queue"
  • Resource Type: Written Reference / Tutorial
  • Access / Price: 100% Free
  • Status: Required
  • Description: Official code templates illustrating how to catch exceptions, forward payload bytes to a DLQ, and resume poll loops.
  • Mutual Exclusivity Mapping: Standard code patterns.

24. Spring Boot Kafka: Advanced Error Handling (Pluralsight)

  • Direct URL: https://www.pluralsight.com/courses/spring-boot-kafka-error-handling
  • Search Identification: Search Pluralsight for: "Spring Boot Kafka Error Handling"
  • Resource Type: Video Course
  • Access / Price: Paid / Free Trial Available
  • Status: Alternative to: Error Handling Patterns in Apache Kafka.
  • Description: Framework-driven guide configuring Spring's DefaultErrorHandler, DeadLetterPublishingRecoverer, and exponential backoff retry listeners.
  • Mutual Exclusivity Mapping: Language-specific alternative.

25. Spring Cloud Stream Resiliency Specs

  • Direct URL: https://spring.io/projects/spring-cloud-stream
  • Search Identification: Search Google/Web for: "Spring Cloud Stream reference manual retry and DLQ"
  • Resource Type: Written Reference / Spec Sheet
  • Access / Price: 100% Free
  • Status: Optional
  • Description: Technical specs detailing configuration properties for automated DLQ routing in Spring Cloud bindings.
  • Mutual Exclusivity Mapping: Framework-specific spec reference.

Sub-Topic F: Zookeeperless Kafka Clustering (KRaft)

26. Zookeeperless Kafka: Clustering with KRaft

  • Direct URL: https://www.linkedin.com/learning/zookeeperless-kafka-clustering-with-kraft
  • Search Identification: Search LinkedIn Learning for: "Zookeeperless Kafka Clustering with KRaft"
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Enterprise Account)
  • Status: Required (Non-Negotiable)
  • Description: Exceptional course detailing how to configure controller quorums, manage metadata nodes, and migrate from ZooKeeper to KRaft.
  • Mutual Exclusivity Mapping: Essential modern clustering guide.

27. Apache Kafka 4.0 Administration and Operations (Udemy)

  • Direct URL: https://www.udemy.com/course/kafka-administration/
  • Search Identification: Search Udemy for: "Apache Kafka Administration and Operations" (Instructor: Stephane Maarek)
  • Resource Type: Video Course
  • Access / Price: Paid (Included in TCS Udemy Business)
  • Status: Required
  • Description: Focused course covering KRaft metadata logs, broker sizing, partition distribution, and cluster upgrades in ZooKeeperless clusters.
  • Mutual Exclusivity Mapping: Essential operations cluster blueprint.

28. Kafka KRaft Mode Official Protocol Specification

29. Operating Zookeeperless Kafka at Scale (Pluralsight)

  • Direct URL: https://www.pluralsight.com/courses/operating-zookeeperless-kafka
  • Search Identification: Search Pluralsight for: "Operating Zookeeperless Apache Kafka"
  • Resource Type: Video Course
  • Access / Price: Paid / Free Trial Available
  • Status: Alternative to: Zookeeperless Kafka: Clustering with KRaft.
  • Description: Cloud-centric course covering broker cluster sizing, memory layouts, and monitoring under KRaft.
  • Mutual Exclusivity Mapping: Choose this if you prefer Pluralsight.

30. Local 3-Broker KRaft Cluster Docker Compose Sandbox

  • Direct URL: https://github.com/conduktor/kafka-stack-docker-compose/blob/master/zk-multiple-kraft.yml
  • Search Identification: Search GitHub for: "conduktor kafka-stack-docker-compose zk-multiple-kraft"
  • Resource Type: Interactive Code Template / Infrastructure-as-Code
  • Access / Price: 100% Free
  • Status: Optional
  • Description: Docker script to spin up a local 3-broker ZooKeeperless cluster with separate controller quorums for cluster crash-testing.
  • Mutual Exclusivity Mapping: Standard local scaling playground.

3. Hands-On Portfolio Lab Project: Transactional DLQ Pipeline

To demonstrate your distributed systems engineering capabilities to product-firm recruiters, you must build and commit a complete Event-Driven Resiliency Pipeline using Python and a local Docker Kafka cluster.

The Lab Project Guidelines:

  1. System Target: You will construct a Payment Processing Stream with three primary elements:
    • Transactions Producer: Emits payment records into the payment-events topic.
    • Payment Processor (Consumer): Reads event payloads, parses payment data, and writes successes.
    • Dead Letter Queue (DLQ) Forwarder: Automatically catches parsing exceptions or transactional validation failures and routes failed records to a separate payment-events-dlq topic.
  2. The Resiliency Requirement:
    • If your consumer encounters a transient error (such as a temporary database timeout), it must not crash or drop the message. It must catch the exception, initiate an Exponential Backoff Retry Loop (e.g., retry 3 times at 1s, 2s, 4s intervals).
    • If all retries fail, or if it encounters a fatal non-transient error (such as corrupt BSON/JSON payload bytes), it must write the failed event to the payment-events-dlq topic along with header tags explaining the failure stack trace, and commit the original offset to keep the pipeline moving.
  3. Algorithmic Architecture:
    • Write a Python consumer using the confluent-kafka or kafka-python library.
    • Implement transactional error-handling loops wrapping the poll context: 
      # Pseudo-code pattern
try:
    payload = json.loads(msg.value())
    process_payment(payload)
    consumer.commit()
except FatalError as e:
    forward_to_dlq(msg, error=str(e))
    consumer.commit() # Keep processing subsequent records
  4. GitHub Commitment: Commit the complete transaction producer (producer.py), payment consumer with DLQ forwarding logic (resilient_consumer.py), a quick docker-compose script that provisions a local broker, and a README.md containing explanation charts to your public 2026-upskilling-roadmap repository.

4. Technical Interview Self-Assessment

Use these questions to verify if you have successfully digested the principles of this event-driven streaming chapter:

ConceptHigh-Frequency Interview QuestionExpected Technical Answer Framework
Kafka vs. RabbitMQWhat is the fundamental structural difference between Apache Kafka and traditional message brokers like RabbitMQ?RabbitMQ is a traditional broker that uses push-based delivery, tracking message consumptions. It removes messages immediately after acknowledgement. The broker is stateful and complex. Kafka is a distributed append-only commit log. Messages are written sequentially to disk and kept based on retention. The broker is stateless ("dumb broker, smart client"); consumers track their own progress using Offsets, allowing multiple groups to read the same stream independently.
Offset CommitsExplain the difference between auto-committing offsets and manual offsets commits, and why auto-commits are risky.Auto-commit (enable.auto.commit=true): The consumer automatically commits its latest offset at staggered intervals (e.g. every 5s). Manual commit: The developer explicitly calls .commit() in the code. Auto-commit is risky because if a consumer pulls 100 messages, commits offset 100 automatically, and then crashes while executing message 20, the promoted consumer resumes from offset 100, silently dropping messages 21 to 99. Manual commits ensure a message is processed before committing.
KRaft vs. ZookeeperWhy did the Apache Kafka community deprecate ZooKeeper in favor of KRaft, and what are the benefits?ZooKeeper maintained metadata outside the Kafka cluster, creating synchronization bottlenecks, slow controller elections, and a split-brain risk. Upgrading metadata involved heavy network replication, limiting Kafka to 200,000 partitions. KRaft incorporates metadata management directly inside select Kafka controller brokers using a Raft consensus log. This eliminates dual-cluster overhead, accelerates controller election speeds, and allows clusters to scale to millions of partitions.

5. Exit Tasks for this Phase

Complete these verification steps before proceeding to Part 11:

  • Launches a local Kafka broker using Docker: docker-compose up -d.
  • Writes and runs the transactions producer script to stream 1,000 mock events.
  • Deploys the payment processor consumer, verifying that corrupt payloads automatically route to the DLQ topic.
  • Commits the resilient consumer script and explain log files to your public Git repository.

Proceed to Part 11: System Design & Scalability Fundamentals →

Comments

Comments are powered by giscus. Set PUBLIC_GISCUS_REPO_ID and PUBLIC_GISCUS_CATEGORY_ID in your environment to enable them.