Apache Kafka Controller

Kafka Controller Nedir?

Açıklaması şöyle

Kafka controller is a thread that runs inside only one broker in a Kafka cluster i.e. If we have a cluster of N brokers then there will be only one broker that is the controller.

It is like a brain for the cluster so that the cluster functions in a smooth and resilient way.

Şeklen şöyle. Burada Controller thread görülebilir.

Kafka Controller Diğer Broker'lardan Farklı Değildir

Açıklaması şöyle

The controller is the same as other brokers except it has some extra responsibilities ...

Kafka Controller Ne Yapar?

Yaptığı temel işler şöyle. Şekilde de görüldüğü gibi diğer broker'ların hayatta olduğunu kontrol eder.

1. Broker Liveliness

2. Leader Election

3. Updating the ISR:

Bu işlerin çoğu aslında Topic için Leader seçmekle ilgili. Açıklaması şöyle

The Controller performs very crucial tasks

Broker Liveliness: The controller performs one of the crucial tasks of monitoring a broker’s liveliness by keeping a watch at the zookeeper znode. Once a broker’s session with zookeeper is lost the controller will consider it as a broker failure. For some of the partitions, this broker failure will be a leader loss, this leadership has to be transferred to another broker.

Leader Election: If a broker fails, then the partitions of which this broker is a leader will require a new leader. The controller will elect a new leader from the ISR list and will persist this mapping in the Zookeeper in order to avoid loss of information. Not only on a failure but also when a new topic is created, the controller will assign leaders for its partition(s) too.

Updating the ISR: Once a new leader for a partition has been elected the controller will persist the same in zookeeper and will broadcast this change to all the brokers(Remember any broker in the cluster can serve the metadata request)When the Controller elects a new leader, it updates the leader epoch and sends that information to all the members of the ISR list and also persists this ISR list in the zookeeper.

Kafka Controller Nasıl Seçilir?

Açıklaması şöyle. Yani zookeeper'da ilk "/controller" düğümünü oluşturan Kafka Controller seçilir.

Whenever a Kafka Cluster is spun up, the brokers will first create a session with the zookeeper and the brokers will try to create an ephemeral node “/controller” inside the zookeeper. The broker that will be able to successfully create the “/controller” node will become the controller.

The rest of the brokers will create a watch on this “/controller” node.

Yeni Kafka Controller Nasıl Seçilir?

Açıklaması şöyle. Yani mevcut Kafka Controller kapatılırsa, zookeeper "/controller" düğümünü siler ve diğer Kafka broker'lar haberdar edilir. Yine aynı işlem tekrarlanır ve ilk "/controller" düğümünü yaratan yeni Kafka Controller seçilir.

Suppose the controller broker goes down Then the cluster should start behaving abnormally? Right? This actually never happens. Kafka is very good when it comes to self-healing. So, to overcome such a scenario rest of the brokers (Kafka Cluster minus Kafka Controller) will create a watch on the “/controller” node. As soon as the controller goes down or its session with the zookeeper is lost then this znode will be deleted and the rest of the brokers will be notified, and a new controller will be elected again.

Yeni Kafka Controller'ın yapması gereken şeyler vardır. Öncelikle zookeeper üzerindeki metadata'yı okuması gerekir. Açıklaması şöyle.

As explained if a controller fails, a new controller will be elected and this new Controller will initialize itself by reading a lot of metadata i.e. the metadata for all Partitions in the cluster, who is the leader, who is the follower, the ISR list, etc. Hence the more the Partitions, the longer this recovery takes.