Apache Cassandra Debezium Connector

Örnek

Şöyle yaparız

{
    "name": "kafka-cosmosdb-sink",
    "config": {
        "connector.class": "com.datastax.oss.kafka.sink.CassandraSinkConnector",
        "tasks.max": "1",
        "topics": "myserver.retail.orders_info",
        "contactPoints": "<Azure Cosmos DB account name>.cassandra.cosmos.azure.com",
        "loadBalancing.localDc": "<Azure Cosmos DB region e.g. Southeast Asia>",
        "datastax-java-driver.advanced.connection.init-query-timeout": 5000,
        "ssl.hostnameValidation": true,
        "ssl.provider": "JDK",
        "ssl.keystore.path": "<path to JDK keystore path e.g. <JAVA_HOME>/jre/lib/security/cacerts>",
        "ssl.keystore.password": "<keystore password: it is 'changeit' by default>",
        "port": 10350,
        "maxConcurrentRequests": 500,
        "maxNumberOfRecordsInBatch": 32,
        "queryExecutionTimeout": 30,
        "connectionPoolLocalSize": 4,
        "auth.username": "<Azure Cosmos DB user name (same as account name)>",
        "auth.password": "<Azure Cosmos DB password>",
        "topic.myserver.retail.orders_info.retail.orders_by_customer.mapping": "order_id=value.orderid, customer_id=value.custid, purchase_amount=value.amount, city=value.city, purchase_time=value.purchase_time",
        "topic.myserver.retail.orders_info.retail.orders_by_city.mapping": "order_id=value.orderid, customer_id=value.custid, purchase_amount=value.amount, city=value.city, purchase_time=value.purchase_time",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "transforms": "unwrap",
        "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
        "offset.flush.interval.ms": 10000
    }
}

Accord Consensus Algorithm

Giriş

Açıklaması şöyle

... called Accord (pdf) from a team at Apple and the University of Michigan. 

Açıklaması şöyle

Accord addresses two problems that aren’t solved in previous consensus protocols: How can we have a globally available consensus and achieve it in one round trip? The first novel mechanism is the reorder buffer.

Assuming commodity hardware is in use, differences in clocks between nodes are inevitable. The reorder buffer measures the difference between nodes in addition to the latency between them. Each replica can use this information to correctly order data from each node and account for the differences, guaranteeing one round-trip consensus with a timestamp protocol.

The other mechanism is fast-path electorates. Failure modes can create latency when electing a new leader before resuming. Fast-path electorates use pre-existing features in Cassandra with some novel implementations to maintain a leaderless fast path to quorum under the same level of failure tolerated by Cassandra.

Apache Kafka Partition Rakamını Hesaplama

Partition Nedir?

Açıklaması şöyle. Topic bölümlendirilir. Her parçaya partition denilir.

Topics are divided into smaller parts called partitions. A partition can be described as a commit log. Messages can be appended to the log and can be read-only in the order from the beginning to the end. Partitions are designed to provide redundancy and scalability. The most important fact is that partitions can be hosted on different servers (brokers), giving a very powerful way to scale topics horizontally.

Partition Yöntemleri

Açıklaması şöyle. Consistent Hashing yazısına bakabilirsiniz.

A common and efficient approach is Consistent Hashing. Although, others such as Hashing and range-based partitioning can still be used.

Global Event Order

Partition ve Message Order yazısına taşıdım

Partition Rakamı Nasıl Belirlenir?

kafka-topics komutu ile topic yaratırken bir de istenilen partition sayısı belirtilebiliyor. Bu sayı nasıl hesaplanır. 

Broker Sayısı

Eğer broker sayısı < 6 ise açıklaması şöyle

Partitions per topic is the million-dollar question and there’s no one answer. If you have a small cluster of fewer than six brokers, create two times the number of brokers you have ( N X 2 if N < 6). The reasoning behind it is that if you have more brokers over time, for example 12 brokers. Well, at least you will have enough partitions to cover that.

Consumer Sayısı

partition değeri ile ilgili açıklama şöyle. Yani consumer sayısı da Partition Rakamını belirlerken bir etken

Remember! Kafka topic partition must be the same or less than the number of concurrent consumer threads

Sonuç

Bu iki sayının en küçüğü alınır

Backend for Frontend - BFF

Giriş

Açıklaması şöyle

Instead of a single point of entry, it introduces multiple gateways. Because of that, you can have a tailored API that targets the needs of each client (mobile, web, desktop, voice assistant, etc.), and remove a lot of the bloat caused by keeping it all in one place.

Şeklen şöyle. Web ve Mobile için farklı giriş noktaları sağlıyor

Aslında bu BFF bir ilk değil. Hemen hemen aynı şeyi anlatan diğer bazı denemeler de şöyle. Bunlar MVC, MVP, MVVM vs.

Apache Kafka Zookeeper Yerine Kafka Raft

Neden Gerekiyor ? 

Açıklaması şöyle. Yani Zookeeper ile her şey yavaş kalıyor. Ayrıca Kafka ve Zookeeper iki farklı araç ve birbirlerinden çok farklı oldukları için her ikisini de öğrenmek gerekiyor. Bu da zaman ve enerji kaybı

In a Kafka cluster, the Kafka Controller manages broker, topic, and partition meta-data (the “control plane”) 

...

But how does Kafka know which controller should be active, and where the meta-data is stored in case of broker failures? Traditionally a ZooKeeper Ensemble (cluster) has managed the controller elections and meta-data storage ...

But this introduces a bottleneck between the active Kafka controller and the ZooKeeper leader, and meta-data changes (updates) and failovers are slow (as all communication is between one Kafka broker and one ZooKeeper server, and ZooKeeper is not horizontally write scalable). The new Kafka Raft mode (KRaft for short) replaces ZooKeeper with Kafka topics and the Raft consensus algorithm to make Kafka self-managed,...

The Kafka cluster meta-data is now only stored in the Kafka cluster itself, making meta-data update operations faster and more scalable. The meta-data is also replicated to all the brokers, making failover from failure faster too. Finally, the active Kafka controller is now the Quorum Leader, using Raft for leader election.

The motivation for giving Kafka a “brain transplant” (replacing ZooKeeper with KRaft) was to fix scalability and performance issues, enable more topics and partitions, and eliminate the need to run an Apache ZooKeeper cluster alongside every Kafka cluster.

Zookeeper'ın Kaldırılması

Kafka 2.8.0'dan itibaren KPI-500 ile (yani Kafka Process Improvement - 500) Zookeper'ın kaldırılması için çalışılıyor. Kafka Version 2.8.0 çıktığında bu hala gerçekleşmemişti. Açıklaması şöyle

Kafka Version 2.8.0 introduces an early access to Zookeeper-less Kafka as part of KPI-500 using the Kraft mode. The implementation is partially complete and thus is not to be used in production environments.

Kafka 3.3 ile Zookeeper isteğe bağlı olarak kaldırılabilir. Açıklaması şöyle

The changes have been happening incrementally, starting from Kafka version 2.8 (released April 2021) which had KRaft in early access. In version 3.0 (September 2021) it was in preview. The maturity of KRaft has improved significantly since version 2.8 (3.0 had 20 KRaft improvements and bug fixes, 3.0.1 had 1, 3.1.0 had 5, and 3.1.1 had only 1 minor fix), and on October 3 2022 version 3.3 was marked as production ready (for new clusters).

Zookeeper Yerine KRaft

Açıklaması şöyle

“KRaft” stands for “Kafka Raft”, and is a combination of Kafka topics and the Raft consensus algorithm.

@metadata Topic

Açıklaması şöyle. Internal Quorum'daki controller'lar artık @metadata topic ile haberleşecekler.

When Kafka Raft Metadata mode is enabled, Kafka will store its metadata and configurations into a topic called @metadata. This internal topic is managed by the internal quorum and replicated across the cluster. The nodes of the cluster can now serve as brokers, controllers or both (called combined nodes).

When KRaft mode is enabled, only a few selected servers can serve as controllers and these will compose the internal quorum. The controllers can either be in active or standby mode that will eventually take over if the current active controller server fails or is taken down.

Normalde "Broker Liveliness" bilgisi Zookeeper'da saklanır. Kafka Controller "liveliness" bilgisini sürekli kontrol eder. Bu artık olmayacak.

AWS Simple Email Service - SES

Giriş

Açıklaması şöyle

Amazon Simple Email Service (SES) is an API for sending emails and managing email lists. SES’s main API endpoints are focused on sending emails and managing email contacts. The service also includes more advanced endpoints related to deliverability, like managing the dedicated IP addresses from which SES sends your emails.

Açıklama şöyle

The first thing you have to do is to add email identities to Amazon SES. It will only allow the registered email identities to send the emails. By default, your Amazon SES will be working on a sandbox environment. So you can only send the emails to the registered email identities. You can request AWS support to remove the sandbox environment.

Dcoker Compose ve NGINX

Örnek

Şöyle yaparız

version: '3'
services:
  lb:
    build:
      context: nginx
      dockerfile: Dockerfile
    ports:
      - "9090:9090"
    networks:
      - my-network
    depends_on:
      - service1
      - service2

  service1:
    build:
      context: service1
      dockerfile: Dockerfile
    ports:
      - "8181:8080"
    networks:
      - my-network

  service2:
    build:
      context: service2
      dockerfile: Dockerfile
    ports:
      - "8282:8080"
    networks:
      - my-network

networks:
  my-network:
    driver: bridge