Apache Kafka Zookeeper

Giriş

Zookeeper Control Plane için gerekir. Açıklaması şöyle

Control and data planes

Apache Kafka implements independent control and data planes for its clusters. The control plane manages the cluster, keeps track of what brokers are alive, and takes action when the set changes. Meanwhile, the data plane consists of the features required to handle producers and consumers and their records. In the previous iterations, Zookeeper was the cluster component that held most of the implementation of the control plane. 

Zookeeper'ın Kaldırılması

KRaft yazısına taşıdım

Zookeeper Instance Sayısı

Tek sayı olması iyi olur. Açıklaması şöyle. Örneğin 3 tane gibi.

We must ensure that the number of Zookeeper nodes is 2n + 1 where n is any number greater than 0. Having an odd number of servers allows Zookeeper to perform majority elections for leadership. This setup can handle n faulty/ dead nodes and any value higher will bring the ensemble down.

Zookeeper Kullanımı

Zookeeper'a bağlanmak için şöyle yaparız

kafka-broker/bin/zookeeper-shell.sh localhost:2181

Docker Compose

Örnek

Şöyle yaparız. Burada "zoonavigator gui" de kullanılıyor.

version: '3.9'
services:

  zoo1:
    container_name: zks1
    image: zookeeper
    restart: always
    hostname: zoo1
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_MY_ID = 1
      - ZOO_SERVERS = server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:
    container_name: zks2
    image: zookeeper
    restart: always
    hostname: zoo2
    ports:
      - '2182:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_MY_ID = 2
      - ZOO_SERVERS = server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoonavigator:
    container_name: zoonavigator
    image: elkozmon/zoonavigator
    ports:
      - 9000:9000

Örnek

Şöyle yaparız. Burada 1 zookeeper ve 1 kafta çalıştırılıyor

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:latest
    container_name: broker
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

zookeeper.properties Dosyası

dataDir Alanı

Örnek şöyle

# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=[Path to data directory]\data

Örnek

Şöyle yaparız

dataDir=/path/to/the/dir/
clientPort=2181
tickTime=2000
initLimit=5
syncLimit=2
maxClientCnxns=60
admin.enableServer=false
server.0=1.2.3.4:2888:3888
server.1=1.2.3.5:2888:3888
server.2=1.2.3.6:2888:3888

Diğer zookeeper sunucularını belirtmek için şöyle yapılıyor

server.<id>=<ip-addr>:2888:3888

brokers/ids dizini

Açıklaması şöyle

The ephemeral nodes i.e. active brokers, take place in the /brokers/ids path.

ls seçeneği

Şöyle yaparız

ls
[admin, brokers, cluster, config, consumers, controller, controller_epoch,
 isr_change_notification, latest_producer_id_blocki log_dir_event_notification, zookeeper]

get seçeneği

Bir düğümün içindeki bilgiyi görmek içindir. Şöyle yaparız

get /controller
{"version":1,"brokerid":0,"timestamp":"1606990912859"}