Glossary

Kafka

TL;DR

Apache Kafka is a distributed streaming platform that enables high-throughput, fault-tolerant processing of real-time data streams for building event-driven applications.

Concept

Apache Kafka is an open-source distributed streaming platform developed by LinkedIn and later donated to the Apache Software Foundation. It’s designed to handle real-time data feeds with high throughput and fault tolerance.

Key components and concepts of Kafka include:

  1. Topics: Categories or feeds to which records are published, similar to tables in databases.

  2. Producers: Applications that publish records to Kafka topics.

  3. Consumers: Applications that subscribe to topics and process the published records.

  4. Brokers: Kafka servers that store data and serve clients, forming a cluster for fault tolerance.

  5. Partitions: Topics are split into partitions for parallel processing and scalability, with each partition being an ordered, immutable sequence of records.

  6. Offsets: Sequential IDs assigned to records that allow consumers to track their position in each partition.

  7. Retention: Kafka persists records for a configurable period, allowing consumers to reprocess data.

Kafka is commonly used for:

  • Real-time stream processing
  • Log aggregation
  • Event sourcing
  • Messaging systems
  • Metrics collection and monitoring
  • Commit logs for distributed systems

Organizations use Kafka to build real-time data pipelines, enable microservices communication, process continuous data streams, and create event-driven architectures. Its distributed nature provides high availability and fault tolerance, making it suitable for mission-critical applications that require reliable data processing.

Related words: Message Queues RabbitMQ