3 Ocak 2023 Salı

Apache Kafka Connect Developer Guide

Giriş
İki tane kavram var
1. Connector
2. Task
Açıklaması şöyle
To copy data between Kafka and another system, users instantiate Kafka Connectors for the systems they want to pull data from or push data to. Connectors come in two flavors: SourceConnectors, which import data from another system, and SinkConnectors, which export data to another system. For example, JDBCSourceConnector would import a relational database into Kafka, and HDFSSinkConnector would export the contents of a Kafka topic to HDFS files.

Implementations of the Connector class do not perform data copying themselves: their configuration describes the set of data to be copied, and the Connector is responsible for breaking that job into a set of Tasks that can be distributed to Kafka Connect workers. Tasks also come in two corresponding flavors: SourceTask and SinkTask. Optionally, the implementation of the Connector class can monitor the data changes of external systems and request task reconfiguration.

30 Aralık 2022 Cuma

OpenTelemetry Java Agent

Giriş
Logları direkt bir backend sunucuya gönderir


Örnek
Uygulamayı çalıştırmak için şöyle yaparız
java -javaagent:opentelemetry-javaagent.jar -jar catalog.jar 
Örnek
Şöyle yaparız
java -javaagent:opentelemetry-javaagent.jar -jar target/*.jar
Örnek
Şöyle yaparız
# 1. Add them to the startup commands

java -javaagent:path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=your-service-name -jar myApplication.jar

# 2. Use JAVA_TOOL_OPTIONS and other environment variables

export JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar"
export OTEL_SERVICE_NAME="your-service-name"
java -jar myApplication.jar
docker-composer.override.otel.yml
Örnek
Şöyle yaparız
curl --create-dirs -O -L --output-dir ./otel \
https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
docker-composer.override.otel.yml şöyledir
#docker-compose.override.otel.yml
version: '3'

services:
  [your-service]:
  volumes:
      - "./otel/opentelemetry-javaagent.jar:/otel/opentelemetry-javaagent.jar"
  environment:
   - JAVA_TOOL_OPTIONS=-javaagent:/otel/opentelemetry-javaagent.jar
   - OTEL_SERVICE_NAME=[your-service]
   - DEPLOYMENT_ENV=DOCKER_LOCAL
  extra_hosts:
        - "host.docker.internal:host-gateway"
Çalıştırmak için şöyle yaparız
docker compose -f docker-compose.yml -f docker-compose.override.otel.yml up -d

SpringBoot
Örnek
Açıklaması şöyle
OpenTelemtry java agent uses the Mapped Diagnostic Context(MDC) to propagate the TraceID and SpanID within the service. You can print the TraceID and SpanID in the log line by extracting the values from MDC in the console pattern. Just add this to yourapplication.yml file:
Şöyle yaparız
logging:
  level:
    root: INFO
  pattern:
    console: '[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%mdc{trace_id}/%mdc{span_id}] [%thread] %-5level %C:%M:%L - %msg%n'
O zaman çıktı şöyle
[2023–04–14 06:53:02.166] [0eef6546864ff6a0129a4b6ce06f7cbf/c3e97a8c1174afd2] [http-nio-8080-exec-7] INFO ...

Exporter Selection
3 tane değişken tanımlanabilir
1. OTEL_TRACES_EXPORTER     Trace exporter to be used. Default “otlp”
2. OTEL_METRICS_EXPORTER    Metrics exporter to be used. Default “otlp”
3. OTEL_LOGS_EXPORTER     Logs exporter to be used. Default “otlp”

OTEL_TRACES_EXPORTER
Trace'lerin hangi formatta gönderileceğini belirtir. otlp, jaeger, zipkin, none gibi değerler alabilir. Varsayılan değer otlp

Console Backend
Örnek
Şöyle yaparızopentelemetry-javaagent-all.jar opentelemetry agent için jar.Servisimizi ismi my-service
OTEL_SERVICE_NAME=my-service \
OTEL_TRACES_EXPORTER=logging \
java -javaagent:./opentelemetry-javaagent.jar \
     -jar target/*.jar
Çıktısı şöyle
//Console output
INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - '/owners/{ownerId}' :
99af87eeaa19b83d014463b046884e56 632e2d2a5931bd17 SERVER [tracer: io.opentelemetry.tomcat-7.0:1.13.1-
alpha] AttributesMap{data={net.transport=ip_tcp, http.target=/owners/11, thread.id=120, http.flavor=1.1,
http.status_code=200, net.peer.ip=0:0:0:0:0:0:0:1, thread.name=http-nio-8080-exec-10,
http.host=localhost:8080, http.route=/owners/{ownerId}, http.user_agent=Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36,
http.method=GET, net.peer.port=50778, http.scheme=http}, capacity=128, totalAddedValues=13}
Jaeger Backend
Jaeger'a özel ayarların açıklaması şöyle
OTEL_EXPORTER_JAEGER_ENDPOINT : Full URL of the Jaeger HTTP endpoint
OTEL_EXPORTER_JAEGER_TIMEOUT : Maximum time (in milliseconds) the Jaeger exporter will wait for each batch export
OTEL_EXPORTER_JAEGER_USER : Username to be used for HTTP basic authentication
OTEL_EXPORTER_JAEGER_PASSWORD : Password to be used for HTTP basic authentication
Örnek
Şöyle yaparız
OTEL_SERVICE_NAME=my-service 
OTEL_TRACES_EXPORTER=jaeger 
OTEL_EXPORTER_JAEGER_ENDPOINT=http://localhost:14250 \
  java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar
Aspecto Backend
Örnek
Şöyle yaparız. Bu sefer OTEL_TRACES_EXPORTER tanımlamaya gerek yok. Varsayılan değer otlp ve Aspecto da bu formatı destekliyor. Sadece OTEL_EXPORTER_OTLP_TRACES_ENDPOINT yeterli
OTEL_SERVICE_NAME=my-service 
OTEL_EXPORTER_OTLP_HEADERS=Authorization={ASPECTO_AUTH}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 \
  java -javaagent:./opentelemetry-javaagent.jar \
       -jar target/*.jar
Lightstep Backend
Lightstep normal opentelemetry-javaagent.jar'ı sarmalayan bir başka jar lightstep-opentelemetry-javaagent.jar isimli bir başka jar sağlıyor
Örnek
Şöyle yaparız. Bu sefer OTEL_TRACES_EXPORTER tanımlamaya gerek yok. Varsayılan değer otlp ve Lightstep de bu formatı destekliyor. Sadece OTEL_EXPORTER_OTLP_TRACES_ENDPOINT yeterli
export LS_ACCESS_TOKEN=your-token
export OTEL_SERVICE_NAME=springboot-demo
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.lightstep.com:443

java -javaagent:lightstep-opentelemetry-javaagent.jar -jar target/*.jar
Örnek
Şöyle yaparız. Bu sefer OTEL_TRACES_EXPORTER tanımlamaya gerek yok. Varsayılan değer otlp ve Lightstep de bu formatı destekliyor. Ayrıca OTEL_EXPORTER_OTLP_ENDPOINT değeri de belirtilmiyor çünkü varsayılan değer tanımlı O da http://localhost:4318.  Yani yerel bilgisayarda Lightstep çalışıyor.
export OTEL_SERVICE_NAME=springboot-demo_jaeger
export LIGHTSTEP_ACCESS_TOKEN="<your_ls_access_token"

java -javaagent:opentelemetry-javaagent.jar -jar target/*.jar
Örnek
Şöyle yaparız
export LS_ACCESS_TOKEN=your-token
export OTEL_SERVICE_NAME=springboot-demo
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.lightstep.com:443

26 Aralık 2022 Pazartesi

Protobuf protoc komutu

Giriş
Söz dizimi şöyle
protoc <options> <filename>
-I seçeneği
Girdi olarak kullanılacak dizini belirtir

--java_out seçeneği
Örnek
Şöyle yaparız
protos=...
javaOut=...

for proto in $(find "$protos" -name "*.proto"); do
  echo "Generating Java code for $proto"

  protoc \
    -I "$ROOT"/protos \
    -I "$ROOT" \
    --java_out="$javaOut" \
    "$proto"

  protoc \
    -I "$ROOT"/protos \
    -I "$ROOT" \
    --grpc-java_out="$javaOut" \
    --plugin=protoc-gen-grpc-java=/usr/local/bin/protoc-gen-grpc-java \
    "$proto"
done
python_out seçeneği
Örnek
Şöyle yaparız. Bulunduğumuz dizindeki employee.proto dosyasını python'a çevirir ve çıktıyı yine bulunduğumuz dizine yazar
protoc -I=. - python_out=. employee.proto

Docker Compose ve Confluent Schema Registry

Image olarak şunlar kullanılır
confluentinc/cp-zookeeper:7.2.1
confluentinc/cp-kafka:7.2.1
confluentinc/cp-schema-registry:7.2.1
confluentinc/cp-enterprise-control-center:7.2.1

cp-server veya cp-kafka
Kafka broker

cp-schema-registry
Schema Registry sunucusu

cp-enterprise-control-center
Açıklaması şöyle
Confluent Control Center

The Confluent Control Center is a web based tool for monitoring the health of the cluster and observing the metrics exported from the broker, as well as metrics gathered from the application consumers and producers. 
Bununla schema yaratma ve yönetme işleri görsel olarak yapılabilir.  http://localhost:9021 adresine gitmek gerekir.

Örnek
Şöyle yaparız
version: "3"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-server:5.4.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: "true"
      CONFLUENT_SUPPORT_CUSTOMER_ID: "anonymous"

  kafka-tools:
    image: confluentinc/cp-kafka:5.4.0
    hostname: kafka-tools
    container_name: kafka-tools
    command: ["tail", "-f", "/dev/null"]
    network_mode: "host"

  schema-registry:
    image: confluentinc/cp-schema-registry:5.4.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181"

  control-center:
    image: confluentinc/cp-enterprise-control-center:5.4.0
    hostname: control-center
    container_name: control-center
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - "9021:9021"
    environment:
      CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
      CONTROL_CENTER_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      CONTROL_CENTER_REPLICATION_FACTOR: 1
      CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
      CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
      CONFLUENT_METRICS_TOPIC_REPLICATION: 1
      PORT: 9021
Örnek
Şöyle yaparız
services:
    zookeeper:
        image: confluentinc/cp-zookeeper:7.2.1
        hostname: zookeeper
        container_name: zookeeper
        ports:
            - "2181:2181"
        environment:
            ZOOKEEPER_CLIENT_PORT: 2181
            ZOOKEEPER_TICK_TIME: 2000

    kafka:
        image: confluentinc/cp-kafka:7.2.1
        hostname: kafka
        container_name: kafka
        depends_on:
            - zookeeper
        ports:
            - "9092:9092"
        environment:
            KAFKA_BROKER_ID: 1
            KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
            KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
            KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
            KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081

    schema-registry:
        image: confluentinc/cp-schema-registry:7.2.1
        hostname: schema-registry
        container_name: schema-registry
        depends_on:
            - zookeeper
            - kafka
        ports:
            - "8081:8081"
        environment:
            SCHEMA_REGISTRY_HOST_NAME: schema-registry
            SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'kafka:29092'
            SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

    control-center:
        image: confluentinc/cp-enterprise-control-center:7.2.1
        hostname: control-center
        container_name: control-center
        depends_on:
            - kafka
            - schema-registry
        ports:
            - "9021:9021"
        environment:
            CONTROL_CENTER_BOOTSTRAP_SERVERS: 'kafka:29092'
            CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
            CONTROL_CENTER_REPLICATION_FACTOR: 1
            CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
            CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
            CONFLUENT_METRICS_TOPIC_REPLICATION: 1
            PORT: 9021


22 Aralık 2022 Perşembe

git diff seçeneği - Show Changes Not Yet Staged

Giriş
Açıklaması şöyle
Show Changes Not Yet Staged
Örnek
Açıklaması şöyle
If you face a situation where you need to share some changes in the code base that you have made with your co-worker but you cannot yet push your changes to the remote repository, you can use the patch command of git. Here is how it works:

After you have made all the changes you can run the following command:

git diff > filename.patch

git diff > filename.patch will create a file that will contain all uncommitted the changes you have made so far in the code base. Then you can simple share this file with your coworker via slack,email,USB etc .

You coworker will simply download you patch file and run the following command:

git apply filename.patch

This would apply all the code changes you have made in the code base on your coworker’s copy of the code.

Örnek
branch ve master arasındaki farkı görmek için şöyle yaparız.
git diff featureXYZ_branch master
Örnek
Şöyle yaparız.
git diff remotes/origin/featureXYZ_branch remotes/origin/master

16 Aralık 2022 Cuma

Debezium JSON Örnekleri

Giriş
Yukarıda schema ile payload için alanlar tanımlıdır. Açıklaması şöyle
before: This field contains the state of the record before the operation. It is optional because it may not be present for all events.
after: This field contains the state of the record after the operation. It is always present for INSERT and UPDATE events.
source: This field contains information about the Debezium connector that generated the event.
op: This field specifies the type of operation performed on the record.
ts_ms: This field specifies the timestamp of the event in milliseconds since the Unix epoch.
transaction: This field contains information about the transaction in which the event occurred. It is optional because it can be null for non-transactional operations.

Read
Örnek
{
  "schema": {
    "type": "struct",
    "fields": [ ... ],
    "optional": false,
    "name": "dbserver1.inventory.customers.Envelope"
  },
  "payload": {
    "before": null,
    "after": {
      "id": 1004,
      "first_name": "Anne",
      "last_name": "Kretchmar",
      "email": "annek@noanswer.org"
    },
    "source": {
      "version": "1.6.1.Final",
      "connector": "mysql",
      "name": "dbserver1",
      "ts_ms": 1630246982521,
      "snapshot": "true",
      "db": "inventory",
      "sequence": null,
      "table": "customers",
      "server_id": 0,
      "gtid": null,
      "file": "mysql-bin.000008",
      "pos": 154,
      "row": 0,
      "thread": null,
      "query": null
    },
    "op": "r",
    "ts_ms": 1630246982521,
    "transaction": null
  }
}
Insert İle Yeni Satır - Create
op alanı c yani CREATE gelir. Payload kısmında before ve after bölümleri var. Bu bölümlerde sütun isimleri var. Yeni satır ise before alanı null gelir.
Örnek
   "payload":{ 
      "before":null,
      "after":{ 
         "id":1005,
         "first_name":"Vlad",
         "last_name":"Mihalcea",
         "email":"vlad@acme.org"
      },
      "source":{ 
         "name":"dbserver1",
         "server_id":223344,
         "ts_sec":1500369632,
         "gtid":null,
         "file":"mysql-bin.000003",
         "pos":364,
         "row":0,
         "snapshot":null,
         "thread":13,
         "db":"inventory",
         "table":"customers"
      },
      "op":"c",
      "ts_ms":1500369632095
   }
}
Update
op alanı u yani UPDATE gelir. Hem before hem de after kısmı doludur
Örnek
{
"payload":{ "before":{ "id":1005, "first_name":"Vlad", "last_name":"Mihalcea", "email":"vlad@acme.org" }, "after":{ "id":1005, "first_name":"Vlad", "last_name":"Mihalcea", "email":"vlad.mihalcea@acme.org" }, "source":{ "name":"dbserver1", "server_id":223344, "ts_sec":1500369929, "gtid":null, "file":"mysql-bin.000003", "pos":673, "row":0, "snapshot":null, "thread":13, "db":"inventory", "table":"customers" }, "op":"u", "ts_ms":1500369929464 } }
Delete
op alanı d yani DELETE gelir. after kısmı null gelir
Örnek
{
    "payload":{ 
      "before":{ 
         "id":1005,
         "first_name":"Vlad",
         "last_name":"Mihalcea",
         "email":"vlad.mihalcea@acme.org"
      },
      "after":null,
      "source":{ 
         "name":"dbserver1",
         "server_id":223344,
         "ts_sec":1500370394,
         "gtid":null,
         "file":"mysql-bin.000003",
         "pos":1025,
         "row":0,
         "snapshot":null,
         "thread":13,
         "db":"inventory",
         "table":"customers"
      },
      "op":"d",
      "ts_ms":1500370394589
   }
}

Docker ve Debezium

Örnek
Şöyle yaparız
> docker run -it \
--name zookeeper \
-p 2181:2181 \
-p 2888:2888 \
-p 3888:3888 
debezium/zookeeper:0.5
 
> docker run -it \
--name kafka \
-p 9092:9092 \
--link zookeeper:zookeeper \
debezium/kafka:0.5
 
> docker run -it \
--name mysql \
-p 3306:3306 \
-e MYSQL_ROOT_PASSWORD=debezium \
-e MYSQL_USER=mysqluser \
-e MYSQL_PASSWORD=mysqlpw 
debezium/example-mysql:0.5
 
> docker run -it \
--name kafka-connect \
-p 8083:8083 \
-e GROUP_ID=1 \
-e CONFIG_STORAGE_TOPIC=my_connect_configs \
-e OFFSET_STORAGE_TOPIC=my_connect_offsets \
--link zookeeper:zookeeper \
--link kafka:kafka \
--link mysql:mysql \
debezium/connect:0.5
 
> docker run -it 
--name kafka-watcher \ 
--link zookeeper:zookeeper \
debezium/kafka:0.5 watch-topic -a -k dbserver1.inventory.customers