![\"\"](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2021\/09\/Java-8-Featured-1024x536.png\")
<\/p>\n<\/p>\n
In this tutorial, we’ll learn the basics of Kafka – the use cases and core concepts anyone should know. We can then find and understand more detailed articles about Kafka<\/a>.<\/p>\n Kafka<\/a> is an open-source stream processing platform developed by the Apache Software Foundation. We can use it as a messaging system to decouple message producers and consumers, but in comparison to “classical” messaging systems like ActiveMQ<\/a>, it is designed to handle real-time data streams and provides a distributed, fault-tolerant, and highly scalable architecture for processing and storing data.<\/p>\n Therefore, we can use it in various use cases:<\/p>\n If we deal with Kafka for the first time, we might like to have a local installation to experience its features. We could get this quickly with the help of Docker.<\/p>\n We download an existing image<\/a> and run a container instance with this command:<\/p>\n This will make the so-called Kafka broker<\/em> available on the host system at port 9092. Now, we would like to connect to the broker using a Kafka client. There are multiple clients that we can use.<\/p>\n The Kafka CLI is part of the installation and is available within the Docker container. We can use it by connecting to the container’s bash.<\/p>\n First, we need to find out the container’s name with this command:<\/p>\n In this sample, the name is awesome_aryabhata<\/em>. We then connect to the bash<\/a> using:<\/p>\n Now, we can, for example, create a topic (we’ll clarify this term later) and list all existing topics with this commands:<\/p>\n The Offset Explorer (formerly: Kafka Tool)<\/a> is a GUI application for managing Kafka. We can download<\/a> and install it quickly. Then, we create a connection and specify the host and port of the Kafka broker:<\/p>\n Then, we can explore the architecture:<\/p>\n The UI for Apache Kafka (Kafka UI)<\/a> is a web UI, implemented with Spring Boot and React, and provided as a Docker container for a simple installation<\/a> with the following command:<\/p>\n We can then open the UI in the browser using http:\/\/localhost:8080<\/a> and define a cluster, as this picture shows:<\/p>\n Because the Kafka broker runs in a different container than the Kafka UI’s backend, it will not have access to localhost:9092<\/em>. We could instead address the host system using host.docker.internal:9092<\/em>, but this is just the bootstrapping URL.<\/p>\n Unfortunately, Kafka itself will return a response that leads to a redirection to localhost:9092 again, which won’t work. If we do not want to configure Kafka (because this would break with the other clients then), we need to create a port forwarding from the Kafka UI’s container port 9092 to the host systems port 9092. The following sketch illustrates the connections:<\/p>\n We can setup this container-internal port forwarding, e.g. using socat<\/em>. We have to install it within the container (Alpine Linux), so we need to connect to the container’s bash with root permissions. So we need these commands, beginning within the host system’s command line:<\/p>\n Unfortunately, we need to run socat<\/em> each time we start the container. Another possibility would be to provide an extension to the Dockerfile<\/a>.<\/p>\n Now, we can specify localhost:9092 as the bootstrap server within the Kafka UI and should be able to view and create topics, as shown below:<\/p>\n We have to add the following Maven dependency<\/a> to our project:<\/p>\n We can then connect to Kafka and consume the messages we produced before:<\/p>\n Of course, there is an integration for the Kafka Client in Spring<\/a>.<\/p>\n We can differentiate Kafka clients into consumers and producers.<\/strong> Producers send messages to Kafka, while consumers receive messages from Kafka. They only receive messages by actively polling from Kafka. Kafka itself is acting in a passive way. This allows each consumer to have its own performance without blocking Kafka.<\/p>\n Of course, there can be multiple producers and multiple consumers at the same time. And, of course, one application can contain both producers and consumers.<\/p>\n Consumers are part of a Consumer Group<\/em> that Kafka identifies by a simple name. Only one consumer of a consumer group will receive the message.<\/strong> This allows scaling out consumers with the guarantee of only-once message delivery.<\/p>\n The following picture shows multiple producers and consumers working together with Kafka:<\/p>\n A message (we can also name it “record<\/em>” or “event<\/em>“, depending on the use case) is the fundamental unit of data that Kafka processes. Its payload can be of any binary format as well as text formats like plain text, Avro<\/a>, XML, or JSON.<\/p>\n Each producer has to specify a serializer to transform the message object into the binary payload format. Each consumer has to specify a corresponding deserializer to transform the payload format back to an object within its JVM. We call these components shortly SerDes<\/em>. There are built-in SerDes<\/a>, but we can implement custom SerDes too.<\/p>\n The following picture shows the payload serialization and deserialization process:<\/p>\n Additionally, a message can have the following optional attributes:<\/p>\n A topic is a logical channel or category to which producers publish messages.<\/strong> Consumers subscribe to a topic to receive messages from in the context of their consumer group.<\/p>\n By default, the retention policy of a topic is 7 days, i.e. after 7 days, Kafka deletes the messages automatically, independent of delivering to consumers or not. We can configure this<\/a> if necessary.<\/p>\n Topics consist of partitions (at least one).<\/strong> To be exact, messages are stored in one partition of the topic. Within one partition, messages get an order number (offset<\/em>). This can ensure that messages are delivered to the consumer in the same order as they were stored in the partition. And, by storing the offsets that a consumer group already received, Kafka guarantees only-once delivery.<\/p>\n By dealing with multiple partitions, we can determine that Kafka can provide both ordering guarantees and load balancing over a pool of consumer processes.<\/strong><\/p>\n One consumer will be assigned to one partition when it subscribes to the topic<\/strong>, e.g. with the Java Kafka client API, as we have already seen:<\/p>\n However, for a consumer, it is possible to choose the partition(s) it wants to poll messages from:<\/p>\n The disadvantage of this variant is that all group consumers have to use this, so automatically assigning partitions to group consumers won’t work in combination with single consumers that connect to a special partition. Also, rebalancing is not possible in case of architectural changes like adding further consumers to the group.<\/p>\n Ideally, we have as many consumers as partitions<\/strong>, so that every consumer can be assigned to exactly one of the partitions, as shown below:<\/p>\n If we have more consumers than partitions, those consumers won’t receive messages from any partition:<\/p>\n If we have fewer consumers than partitions, consumers will receive messages from multiple partitions, which conflicts with optimal load balancing:<\/p>\n Producers do not necessarily send messages to only one partition.<\/strong> Every produced message is assigned to one partition automatically, following these rules:<\/p>\n Again, storing messages to the same partition will retain the message ordering, while storing messages to different partitions will lead to disordering but parallel processing.<\/strong><\/p>\n If the default partitioning does not match our expectations, we can simply implement a custom partitioner. Therefore, we implement the Partitioner<\/a> interface and register it during the initialization of the producer:<\/p>\n The following picture shows producers and consumers and their connections to the partitions:<\/p>\n Each producer has its own partitioner, so if we want to ensure that messages are partitioned consistently within the topic, we have to ensure that the partitioners of all producers work the same way, or we should only work with a single producer.<\/p>\n Partitions store messages in the order they arrive at the Kafka broker. Typically, a producer does not send each message as a single request, but it will send multiple messages within a batch. If we need to ensure the order of the messages and only-once delivery within one partition, we need transaction-aware producers and consumers<\/a>.<\/p>\n As we have found out, Kafka uses topic partitions to allow parallel message delivery and load balancing of consumers. But Kafka itself must be scalable and fault-tolerant. So we typically do not use a single Kafka Broker, but a Cluster<\/em> of multiple brokers.<\/strong> These brokers do not behave completely the same, but each of them is assigned special tasks that the rest of the cluster can then absorb if one broker fails.<\/p>\n To understand this, we need to expand our understanding of topics. When creating a topic, we not only specify the number of partitions but also the number of brokers that jointly manage the partitions using synchronization. We call this the Replication Factor<\/em>. For example, using the Kafka CLI, we could create a topic with 6 partitions, each of them synchronized on 3 brokers:<\/p>\n For example, a replication factor of three means, that the cluster is resilient for up to two replica failures (N-1 resiliency<\/em>). We have to ensure that we have at least as many brokers as we specify as the replication factor.<\/strong> Otherwise, Kafka does not create the topic until the count of brokers increases.<\/p>\n For better efficiency, replication of a partition only occurs in one direction.<\/strong> Kafka achieves this by declaring one of the brokers as the Partition Leader<\/em>. Producers only send messages to the partition leader, and the leader then synchronizes with the other brokers. Consumers will also poll from the partition leader because the increasing consumer group’s offset has to be synchronized too.<\/p>\n Partition leading is distributed to multiple brokers.<\/strong> Kafka tries to find different brokers for different partitions. Let’s see an example with four brokers and two partitions with a replication factor of three:<\/p>\n Broker 1 is the leader of Partition 1, and Broker 4 is the leader of Partition 2. So each client will connect to those brokers when sending or polling messages from these partitions. To get information about the partition leaders and other available brokers (metadata), there is a special bootstrapping mechanism. In summary, we can say that every broker can provide the cluster’s metadata, so the client could initialize the connection with each of these brokers, and will redirect to the partition leaders then. That’s why we can specify multiple brokers as bootstrapping servers<\/a>.<\/p>\n If one partition-leading broker fails, Kafka will declare one of the still-working brokers as the new partition leader. Then, all clients have to connect to the new leader. In our example, if Broker 1 fails, Broker 2 becomes the new leader of Partition 1. Then, the clients that were connected to Broker 1 have to switch to Broker 2.<\/p>\n Kafka uses Kraft (in earlier versions: Zookeeper)<\/a> for the orchestration of all brokers within the cluster.<\/p>\n If we put producers and consumers together with a cluster of three brokers that manage a single topic with three partitions and a replication factor of 3, we’ll get this architecture:<\/p>\n We already know that multiple clients like a CLI, a Java-based client with integration to Spring applications, and multiple GUI tools are available to connect with Kafka. Of course, there are further Client APIs for other programming languages (e.g., C\/C++<\/a>, Python<\/a>, or Javascript<\/a>), but those are not part of the Kafka project.<\/p>\n Built on top of these APIs, there are further APIs for special purposes.<\/p>\n Kafka Connect<\/a> is an API for exchanging data with third-party systems. There are existing connectors<\/a> e.g. for AWS S3, JDBC, or even for exchanging data between different Kafka clusters. And of course, we can write custom connectors too.<\/p>\n Kafka Streams<\/a> is an API for implementing stream processing applications that get their input from a Kafka topic, and store the result in another Kafka topic.<\/p>\n KSQL is an SQL-like interface built on top of Kafka Streams. It does not require us to develop Java code, but we can declare SQL-like syntax to define stream processing of messages that are exchanged with Kafka. For this, we use the ksqlDB<\/a>, which connects to the Kafka cluster. We can access ksqlDB with a CLI or with a Java client application.<\/p>\n The Kafka REST proxy<\/a> provides a RESTful interface to a Kafka cluster. This way, we do not need any Kafka clients and avoid using the native Kafka protocol. It allows web frontends to connect with Kafka and makes it possible to use network components like API gateways or firewalls.<\/p>\n Strimzi<\/a> is an open-source project that provides a way to run Kafka on Kubernetes and OpenShift platforms. It introduces custom Kubernetes resources making it easier to declare and manage Kafka-related resources in a Kubernetes-native way. It follows the Operator Pattern<\/a>, i.e. operators automate tasks like provisioning, scaling, rolling updates, and monitoring of Kafka clusters.<\/p>\n In this article, we have learned that Kafka is designed for high scalability and fault tolerance. Producers collect messages and send them in batches, topics are divided into partitions to allow parallel message delivery and load balancing of consumers, and replication is done over multiple brokers to ensure fault tolerance.<\/p>\n As usual, all the code implementations are available\u00a0over on GitHub<\/a>.<\/p>\n <\/p><\/div>\n\n \u7acb\u523b\u8a3b\u518a\u53ca\u5831\u540d\u96fb\u8166\u88dc\u7fd2\u8ab2\u7a0b\u5427! <\/a> www.ITSec.vip<\/a> www.Sraa.com.hk<\/a> www.ITSec.hk<\/a> www.Penetrationtest.hk<\/a> 2. What Is Kafka?<\/h2>\n
<\/div>\n\n
3. Setup A Local Environment<\/h2>\n
<\/div>\n3.1. Install Kafka<\/h3>\n
<\/div>\ndocker run -p 9092:9092 -d bashj79\/kafka-kraft<\/code><\/pre>\n3.2. Use Kafka CLI<\/h3>\n
<\/div>\ndocker ps\r\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\r\n7653830053fa bashj79\/kafka-kraft "\/bin\/start_kafka.sh" 8 weeks ago Up 2 hours 0.0.0.0:9092->9092\/tcp awesome_aryabhata<\/code><\/pre>\ndocker exec -it awesome_aryabhata \/bin\/bash<\/code><\/pre>\ncd \/opt\/kafka\/bin\r\n# create topic 'my-first-topic'\r\nsh kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-first-topic --partitions 1 --replication-factor 1\r\n# list topics\r\nsh kafka-topics.sh --bootstrap-server localhost:9092 --list\r\n# send messages to the topic\r\nsh kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-first-topic\r\n>Hello World\r\n>The weather is fine\r\n>I love Kafka<\/code><\/pre>\n3.3. Use Offset Explorer<\/h3>\n
<\/div>\n![\"Screenshot](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/offsetexplorer1.png\")
<\/p>\n![\"Kafka](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/offsetexplorer2.png\")
<\/p>\n3.4. Use UI for Apache Kafka (Kafka UI)<\/h3>\n
<\/div>\ndocker run -it -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true provectuslabs\/kafka-ui<\/code><\/pre>\n![\"Screenshot](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/kafka-ui.png\")
<\/p>\n![\"Sketch](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/kafka-ui-port-forwarding.png\")
<\/p>\n# Connect to the container's bash (find out the name with 'docker ps')\r\ndocker exec -it --user=root <name-of-kafka-ui-container> \/bin\/sh\r\n# Now, we are connected to the container's bash.\r\n# Let's install 'socat'\r\napk add socat\r\n# Use socat to create the port forwarding\r\nsocat tcp-listen:9092,fork tcp:host.docker.internal:9092\r\n# This will lead to a running process that we don't kill as long as the container's running<\/code><\/pre>\n![\"Topics](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/kafka-ui-topics-1.png\")
<\/p>\n3.5. Use Kafka Java Client<\/h3>\n
<\/div>\n<dependency>\r\n <groupId>org.apache.kafka<\/groupId>\r\n <artifactId>kafka-clients<\/artifactId>\r\n <version>3.5.1<\/version>\r\n<\/dependency><\/code><\/pre>\n\/\/ specify connection properties\r\nProperties props = new Properties();\r\nprops.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");\r\nprops.put(ConsumerConfig.GROUP_ID_CONFIG, "MyFirstConsumer");\r\nprops.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());\r\nprops.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());\r\n\/\/ receive messages that were sent before the consumer started\r\nprops.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");\r\n\/\/ create the consumer using props.\r\ntry (final Consumer<Long, String> consumer = new KafkaConsumer<>(props)) {\r\n \/\/ subscribe to the topic.\r\n final String topic = "my-first-topic";\r\n consumer.subscribe(Arrays.asList(topic));\r\n \/\/ poll messages from the topic and print them to the console\r\n consumer\r\n .poll(Duration.ofMinutes(1))\r\n .forEach(System.out::println);\r\n}<\/code><\/pre>\n4. Basic Concept<\/h2>\n
<\/div>\n4.1. Producers & Consumers<\/h3>\n
<\/div>\n![\"Producers](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/concepts-01-consumers-producers.png\")
<\/p>\n4.2. Messages<\/h3>\n
<\/div>\n![\"Payload](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/concepts-02-records.png\")
<\/p>\n\n
4.3. Topics & Partitions<\/h3>\n
<\/div>\nString topic = "my-first-topic";\r\nconsumer.subscribe(Arrays.asList(topic));<\/code><\/pre>\nTopicPartition myPartition = new TopicPartition(topic, 1);\r\nconsumer.assign(Arrays.asList(myPartition));\r\n<\/code><\/pre>\n![\"ideal](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/concepts-03-partitions-1.png\")
<\/p>\n![\"too](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/concepts-03-partitions-2.png\")
<\/p>\n![\"too](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/concepts-03-partitions-3.png\")
<\/p>\n\n
Properties producerProperties = new Properties();\r\n\/\/ ... \r\nproducerProperties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, MyCustomPartitioner.class.getName());\r\nKafkaProducer<String, String> producer = new KafkaProducer<>(producerProperties);<\/code><\/pre>\n![\"producers](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/concepts-03-partitions-4.png\")
<\/p>\n4.4. Clusters and Partition Replicas<\/h3>\n
<\/div>\nsh kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-replicated-topic --partitions 6 --replication-factor 3<\/code><\/pre>\n![\"Image](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/kafka-topology-1024x549.png\")
<\/p>\n![\"Kafka](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/kafka-topology-failure-1024x581.png\")
<\/p>\n4.4. Putting All Together<\/h3>\n
<\/div>\n![\"Topics,](\"https:\/\/www.baeldung.com\/wp-content\/uploads\/2023\/09\/kafta-topology-1024x408.png\")
<\/p>\n5. Ecosystem<\/h2>\n
<\/div>\n5.1. Kafka Connect API<\/h3>\n
<\/div>\n5.2. Kafka Streams API<\/h3>\n
<\/div>\n5.3. KSQL<\/h3>\n
<\/div>\n5.4. Kafka REST Proxy<\/h3>\n
<\/div>\n5.5. Kafka Operators for Kubernetes (Strimzi)<\/h3>\n
<\/div>\n6. Conclusion<\/h2>\n
<\/div>\n<\/p>\n
![](\"https:\/\/assets.feedblitz.com\/i\/fblike20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/pinterest20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/twitter20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/email20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/rss20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/comments20.png\")
<\/a> ![](\"https:\/\/assets.feedblitz.com\/i\/commentsrss20.png\")
<\/a><\/p>\nRelated Stories<\/h3>\n
\n
Commercials Cooperation Advertisements:<\/b><\/h2>\r\n
(1) IT Teacher IT Freelance
<\/p>\r\n
<\/a>
\r\n\r\n\u7535\u5b50\u8ba1\u7b97\u673a -\u6559\u80b2 -IT \u96fb\u8166\u73ed\u201d ( IT\u96fb\u8166\u88dc\u7fd2 ) \u63d0\u4f9b\u4e00\u500b\u65b9\u4fbf\u7684\u7535\u5b50\u8ba1\u7b97\u673a \u6559\u80b2\u5e73\u53f0, \u70ba\u5927\u5bb6\u914d\u5c0d\u4fe1\u606f\u6280\u672f, \u96fb\u8166 \u8001\u5e2b, IT freelance \u548c programming expert. \u8b93\u5927\u5bb6\u65b9\u4fbf\u5730\u5c31\u80fd\u627e\u5230\u5408\u9069\u7684\u96fb\u8166\u88dc\u7fd2, \u96fb\u8166\u73ed, \u5bb6\u6559, \u79c1\u4eba\u8001\u5e2b.
\r\n\r\nWe are a education and information platform which you can find a IT private tutorial teacher or freelance.
\r\n\r\nAlso we provide different information about information technology, Computer, programming, mobile, Android, apple, game, movie, anime, animation\u2026 \r\n<\/p>\n
(2) ITSec
<\/p>
<\/a> \r\n
\r\n
\r\n
\r\n
\r\n
\r\n