Docker Compose ve Debezium

Giriş

Debezium aslında 4 tane şeyin aynı anda çalışması ile oluşan bir şey. Bunlar

1. zookeeper

2. kafka

3. kafka-connect

4. debezium

Image olarak şunlar kullanılabilir

debezium/server

debezium/connect

outbox-transformer

Şu değişkenler belirtilir

BOOTSTRAP_SERVERS : Kafka adresi

GROUP_ID

CONFIG_STORAGE_TOPIC

OFFSET_STORAGE_TOPIC

STATUS_STORAGE_TOPIC

Ayrıca şu edeğişkenler ile Avro sunucusu belirtilir

KEY_CONVERTER

VALUE_CONVERTER

CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL

CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL

Örnek - postgres + debezium + kafka

Şöyle yaparız. 

version: '3.1'
services:
    postgres:
        ...
    zookeeper:
        ...
    kafka:
        ...
    connector:
        image: debezium/connect:latest
        ports:
          - "8083:8083"
        environment:
          GROUP_ID: 1
          CONFIG_STORAGE_TOPIC: my_connect_configs
          OFFSET_STORAGE_TOPIC: my_connect_offsets
          BOOTSTRAP_SERVERS: kafka:9092
        depends_on:
          - zookeeper
          - postgres
          - kafka

Örnek - postgres + debezium + kafka

Şöyle yaparız

services:
  db:
    ...

  zookeeper:
    ...

  kafka:
    ...

  connect:
    image: debezium/connect
    ports:
      - "8083:8083"
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses
    depends_on:
      - zookeeper
      - kafka

Örnek - postgres + debezium + kafka

Şöyle yaparız

version: "3.5"

services:
  # Install postgres and setup the user service database
  postgres:
    ...

  # Install zookeeper.
  zookeeper:
   ...

  # Install kafka and create needed topics.
  kafka:
    ...

  # Install debezium-connect and add outbox-transformer here.
  debezium-connect:
    container_name: custom-debezium-connect
    image: outbox-transformer
    hostname: debezium-connect
    ports:
      - '8083:8083'
    environment:
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: debezium_connect_config
      OFFSET_STORAGE_TOPIC: debezium_connect_offsets
      STATUS_STORAGE_TOPIC: debezium_connect_status
      BOOTSTRAP_SERVERS: kafka:29092
    depends_on:
      - kafka
      - postgres

Örnek - postgres + debezium + kafka + avro

Şöyle yaparız

version: “3.7”
services:
  postgres:
   ...
  zookeeper:
   ...
  kafka:
    ...
  kafka-ui:
   ...
  debezium:
    image: debezium/connect:1.4
    environment:
    BOOTSTRAP_SERVERS: kafka:9092
    GROUP_ID: 1
    CONFIG_STORAGE_TOPIC: connect_configs
    OFFSET_STORAGE_TOPIC: connect_offsets
    KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
    VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
    CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
    CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
    depends_on: [kafka]
    ports:
      — 8083:8083
  schema-registry:
   ...

Örnek

Şöyle yaparız. Burada Debezium Kafka yerine Redis'e yazıyor. Connector ayarları conf dizinindeki application.properties dosyasında

version: '3.1'
services:
  redis:
    image: redis
    ports:
      - 6379:6379
    depends_on:
      - postgres
  postgres:
    image: postgres
    restart: always
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - ./postgresql.conf:/etc/postgresql/postgresql.conf
      - ./init:/docker-entrypoint-initdb.d
    command:
      - "-c"
      - "config_file=/etc/postgresql/postgresql.conf"
    ports:
      - 5432:5432
  debezium:
    image: debezium/server
    volumes:
      - ./conf:/debezium/conf
      - ./data:/debezium/data
    depends_on:
      - redis