Apache Kafka Zookeeper Yerine Kafka Raft

Neden Gerekiyor ? 

Açıklaması şöyle. Yani Zookeeper ile her şey yavaş kalıyor. Ayrıca Kafka ve Zookeeper iki farklı araç ve birbirlerinden çok farklı oldukları için her ikisini de öğrenmek gerekiyor. Bu da zaman ve enerji kaybı

In a Kafka cluster, the Kafka Controller manages broker, topic, and partition meta-data (the “control plane”) 

...

But how does Kafka know which controller should be active, and where the meta-data is stored in case of broker failures? Traditionally a ZooKeeper Ensemble (cluster) has managed the controller elections and meta-data storage ...

But this introduces a bottleneck between the active Kafka controller and the ZooKeeper leader, and meta-data changes (updates) and failovers are slow (as all communication is between one Kafka broker and one ZooKeeper server, and ZooKeeper is not horizontally write scalable). The new Kafka Raft mode (KRaft for short) replaces ZooKeeper with Kafka topics and the Raft consensus algorithm to make Kafka self-managed,...

The Kafka cluster meta-data is now only stored in the Kafka cluster itself, making meta-data update operations faster and more scalable. The meta-data is also replicated to all the brokers, making failover from failure faster too. Finally, the active Kafka controller is now the Quorum Leader, using Raft for leader election.

The motivation for giving Kafka a “brain transplant” (replacing ZooKeeper with KRaft) was to fix scalability and performance issues, enable more topics and partitions, and eliminate the need to run an Apache ZooKeeper cluster alongside every Kafka cluster.

Zookeeper'ın Kaldırılması

Kafka 2.8.0'dan itibaren KPI-500 ile (yani Kafka Process Improvement - 500) Zookeper'ın kaldırılması için çalışılıyor. Kafka Version 2.8.0 çıktığında bu hala gerçekleşmemişti. Açıklaması şöyle

Kafka Version 2.8.0 introduces an early access to Zookeeper-less Kafka as part of KPI-500 using the Kraft mode. The implementation is partially complete and thus is not to be used in production environments.

Kafka 3.3 ile Zookeeper isteğe bağlı olarak kaldırılabilir. Açıklaması şöyle

The changes have been happening incrementally, starting from Kafka version 2.8 (released April 2021) which had KRaft in early access. In version 3.0 (September 2021) it was in preview. The maturity of KRaft has improved significantly since version 2.8 (3.0 had 20 KRaft improvements and bug fixes, 3.0.1 had 1, 3.1.0 had 5, and 3.1.1 had only 1 minor fix), and on October 3 2022 version 3.3 was marked as production ready (for new clusters).

Zookeeper Yerine KRaft

Açıklaması şöyle

“KRaft” stands for “Kafka Raft”, and is a combination of Kafka topics and the Raft consensus algorithm.

@metadata Topic

Açıklaması şöyle. Internal Quorum'daki controller'lar artık @metadata topic ile haberleşecekler.

When Kafka Raft Metadata mode is enabled, Kafka will store its metadata and configurations into a topic called @metadata. This internal topic is managed by the internal quorum and replicated across the cluster. The nodes of the cluster can now serve as brokers, controllers or both (called combined nodes).

When KRaft mode is enabled, only a few selected servers can serve as controllers and these will compose the internal quorum. The controllers can either be in active or standby mode that will eventually take over if the current active controller server fails or is taken down.

Normalde "Broker Liveliness" bilgisi Zookeeper'da saklanır. Kafka Controller "liveliness" bilgisini sürekli kontrol eder. Bu artık olmayacak.