apache kafka Training

This course focuses on designing reliable event-driven systems with Apache Kafka. It covers both Kafka internals and architectural decisions, with practical labs and production best practices.

Apache Kafka Essentials

Duration: 2 days

Audience: Architects, developers, technical leads, DevOps engineers

Format: On-site or remote

Download Course Outline (PDF)

Why this course?
  • Designed for engineers building production systems
  • Based on real consulting experience
  • Hands-on exercises throughout the course
  • Focus on architecture and system design rather than programming
  • No coding is required during the course
  • Updated regularly to reflect the latest Kafka features and best practices
Day 1. Morning Module. Kafka Fundamentals and Client APIs

Part 1: Kafka Overview

  • Kafka ecosystem
  • Use cases
  • Messaging concepts
  • Topics and partitions
  • Replication
  • KRaft overview

Part 2: Producers

  • Producer architecture
  • Partition selection
  • Keys and ordering
  • Acknowledgements
  • Retries
  • Batching and compression
  • Producer troubleshooting

Part 3: Consumers

  • Consumer groups
  • Offsets
  • Rebalancing
  • Group coordinator
  • New Kafka 4.0 consumer protocol
  • Positioning consumers
  • Consumer troubleshooting

Part 4: Storage

  • Retention
  • Segment files
  • Compaction
  • State vs event topics
  • GDPR considerations

Lab

  • Create topics
  • Produce and consume records
  • Observe partition assignment
Day 1. Afternoon Module. Kafka in context, Schema Registry and Security

Part 1: Kafka in context

  • When Kafka fits
  • Anti-patterns
  • Kafka vs Database
  • Kafka vs RabbitMQ
  • Kafka vs Pulsar
  • Kafka Queues (Kafka 4.0)

Part 2: Schema Registry

  • Why schemas matter
  • Avro fundamentals
  • Schema Registry architecture
  • Compatibility modes
  • Schema evolution
  • Broker-side schema validation

Part 3: Cluster Architecture

  • ZooKeeper architecture
  • Controller election
  • KRaft architecture
  • KRaft migration process
  • Listener configuration

Part 4: Security

  • TLS/SSL
  • SASL mechanisms
  • ACL authorization
  • End-to-end encryption

Lab

  • Avro producer/consumer
  • Schema evolution exercise
Day 2. Morning Module. Reliability, Performance, and Kafka Connect

Part 1: Reliability

  • Replication internals
  • ISR
  • Acknowledgement strategies
  • Idempotent producers
  • Transactions
  • Exactly-once processing

Part 2: Performance Tuning

  • Throughput optimization
  • Latency optimization
  • Durability optimization
  • Availability optimization

Part 3: Kafka Connect

  • Connect architecture
  • Workers and tasks
  • Converters
  • SMTs
  • JDBC connectors
  • Debezium CDC

Part 4: REST Proxy

  • Architecture
  • Use cases
  • Trade-offs

Lab

  • JDBC Source Connector
  • JDBC Sink Connector
Day 2. Afternoon Module. Multi-Cluster Architectures, Monitoring and Stream Processing

Part 1: Multi-Cluster Kafka

  • Active-Passive
  • Active-Active
  • Stretched Cluster
  • RPO/RTO concepts

Part 2: Monitoring

  • Broker metrics
  • Producer metrics
  • Consumer metrics
  • Prometheus
  • Cruise Control,  Kafka Monitor, Trogdor

Part 3: Stream Processing Concepts

  • Event-driven analytics
  • Stateless transformations
  • Stateful transformations
  • Event time
  • Windowing

Part 4: Kafka Streams

  • Streams architecture
  • KStream
  • KTable
  • State stores
  • Joins
  • Aggregations
  • Scaling and parallelism