26 Aralık 2021 Pazar

Kafka Producer API

Giriş
Açıklaması şöyle.
The Producer API allows an application to publish a stream of records to one or more Kafka topics.
Java örneklerini Kafka Producer API yazısına taşıdım

Idempotent Producer Configuration
Açıklaması şöyle
By configuring the Producer to be idempotent, each Producer is assigned a unique Id (PID) and each message is given a monotonically increasing sequence number. The broker tracks the PID + sequence number combination for each partition, rejecting any duplicate write requests it receives.
enable.idempotence Alanı
Açıklaması şöyle. Yani bu alana ilave olarak acks = all olmalı 
enable.idempotence determines whether the producer may write duplicates of a retried message to the topic partition when a retryable error is thrown. Examples of such transient errors include leader not available and not enough replicas exceptions. It only applies if the retries configuration is greater than 0 (which it is by default).

To ensure the idempotent behaviour then the Producer configuration acks must also be set to all. The leader will wait until at least the minimum required number of in-sync replica partitions have acknowledged receipt of the message before itself acknowledging the message. The minimum number is based on the configuration parameter min.insync.replicas.

By configuring acks equal to all it favours durability and deduplicating messages over performance. The performance hit is usually considered insignificant.
enable.idempotence İçin Timeout
Açıklaması şöyle
The recommendation is to leave the retries as the default (the maximum integer value) and limit retries by time, using the Producer configuration delivery.timeout.ms (defaulted to 2 minutes).
Yani önerilen ayarlar şekle şöyle



KafkaProducer Sınıfı Açısından acks Nedir?
Açıklaması şöyle
For data durability, the KafkaProducer has the configuration setting acks. The acks configuration specifies how many acknowledgments the producer receives to consider a record delivered to the broker. The options to choose from are:

none: The producer considers the records successfully delivered once it sends the records to the broker. This is basically “fire and forget.”
one: The producer waits for the lead broker to acknowledge that it has written the record to its log.
all: The producer waits for an acknowledgment from the lead broker and from the following brokers that they have successfully written the record to their logs.

As you can see, there is a trade-off to make here — and that’s by design because different applications have different requirements. You can opt for higher throughput with a chance for data loss, or you may prefer a very high data durability guarantee at the expense of lower throughput. 
acks=all Durumu
Açıklaması şöyle. min.insync.replicas değerini de atamak gerekir. Bundan sonra NotEnoughReplicasExceptionNotEnoughReplicasAfterAppendException gibi exception'lar alınabilir.
If you produce records with acks set to all to a cluster of three Kafka brokers, it means that, under ideal conditions, Kafka contains three replicas of your data; one for the lead broker and one each for two followers. When the logs of each of these replicas all have the same record offsets, they are considered to be in sync. In other words, these in-sync replicas have the same content for a given topic partition. 

But there’s some subtlety to using the acks=all configuration. What it doesn’t specify is how many replicas need to be in sync. The lead broker will always be in sync with itself. But, you could have a situation where the two following brokers can’t keep up due to network partitions, record load, etc. So, when a producer has a successful send, the actual number of acknowledgments could have come from only one broker! If the two followers are not in sync, the producer still receives the required number of acks, but it’s only the leader in this case.

By setting acks=all, you are placing a premium on the durability of your data. So, if the replicas aren’t keeping up, it stands to reason that you want to raise an exception for new records until the replicas are caught up. In a nutshell, having only one in sync replica follows the "letter of the law" but not the "spirit of the law." What we need is a guarantee when using the acks=all setting. A successful send involves at least a majority of the available in sync brokers. There just so happens to be one such configuration: min.insync.replicas. The min.insync.replicas configuration enforces the number of replicas that must be in sync for the write to proceed. Note that the min.insync.replicas configuration is set at the broker or topic level and is not a producer configuration. The default value for min.insync.replicas is one. So, to avoid the scenario described above, in a three-broker cluster, you’d want to increase the value to two. 

ProducerRecord Sınıfı
Açıklaması şöyle. Sticky Partitioner kullanma imkanı var. Örnek ver
Kafka uses partitions to increase throughput and spread a load of messages to all brokers in a cluster. Kafka records are in a key/value format, where the keys can be null. Kafka producers don’t immediately send records, instead, they place them into partition-specific batches to be sent later. Batches are an effective means of increasing network utilization. There are three ways the partitioner determines into which partition the records should be written. The partition can be explicitly provided in the ProducerRecord object via the overloaded ProducerRecord constructor. In this case, the producer always uses this partition. If no partition is provided, and the ProducerRecord has a key, the producer takes the hash of the key modulo as the number of partitions. The resulting number from that calculation is the partition that the producer will use. Previously, if there was no key and no partition present in the ProducerRecord, Kafka would use a round-robin approach to assign messages across partitions. The producer would assign the first record in the batch to partition zero, the second to partition one, and so on, until the end of the partitions. The producer would then start over with partition zero and repeat the entire process for all remaining records. 

The round-robin approach works well for even distribution of records across partitions. But there’s one drawback. Due to this "fair" round-robin approach, you can end up sending multiple sparsely populated batches. It’s more efficient to send fewer batches with more records in each batch. Fewer batches mean less queuing of produce requests, hence less load on the brokers.

25 Aralık 2021 Cumartesi

Yazılım Mimarisi - Deduplication Patterns

Giriş
Dağıtık mimarilerde ve özellikle microservice mimarisinde aynı mesaj birden fazla kez gelebilir. Bu mesajların ayıklanmasına deduplication deniliyor. Kullanılabilecek bazı yöntemler şöyle

1. Idempotent Consumer Pattern
Idempotency (Denkgüçlülük) Nedir yazısına taşıdım. Ancak kısaca tüketen taraf 
- ya mesajları bir sayı ile takip eder ve tekrar gelen mesajları ayıklar
- ya da tüm işleri en baştan yapar

2. Transactional Outbox
Outbox Pattern yazısına taşıdım. Ancak kısaca üreten ve tüketen taraflar birbirlerini direkt çağırmazlar. Bunun yerine aracı olarak veri tabanını kullanırlar. Böylece 
- üreten taraf veri tabanına yazmayı başardığında mesajın gideceğini garanti eder.
- tüketen taraf mesajı başarıyla işleyip veri tabanından sildiğinde mesajın tüketildiğini garanti eder
- tüketen taraf için kod yazmak şart değil. CDC araçlarından birisi de kullanılabilir

3. Kafka Transaction API — for exactly-once delivery semantics
Bu çözüm tabii ki eğer Kafka kullanıyorsak geçerli. Apache Kafka Message Delivery Semantics yazısına taşıdım. Aslında Transactional Outbox ile aynı mantık. Farklı olarak veri tabanı yerine Kafka kullanılıyor. Böylece
- üreten taraf Kafka Transaction'ınını başarıyla commit'lerse mesajın gideceğini garanti eder.
- tüketen taraf mesajı başarıyla işleyip Kafka Transaction'ınını başarıyla commit'lerse mesajın tüketildiğini garanti eder

19 Aralık 2021 Pazar

git config seçeneği

Giriş
--local, --global, --system seçeneği ile kullanılır

alias.ci
user.name
user.email
core.editor
credential.helper
gibi şeyleri atamak mümkün

user.name ve user.email
Örnek
Author vermek için şöyle yaparız
git config --local user.name "John Doe"
git config --local user.email john@doe.com
Örnek
Şöyle yaparız.
$ git config --global user.name new
$ git config --global user.email new@new.com
alias
Örnek
Alias tanımlamak için şöyle yaparız. "git commit" yerine "git ci" yazabiliriz.
git config --global alias.ci commit
git config --global alias.co checkout
...
core.editor
Örnek
Editörü vi'dan nanoya değiştirmek için şöyle yaparız
$ git config --global core.editor "nano -w"
Örnek
Editörü notepad için şöyle yaparız
$ git config --global core.editor notepad
core.sshCommand
Örnek
Şöyle yaparız
git config core.sshCommand “ssh -i ~/.ssh/id_rsa”

credential.helper
Örnek
Şifreyi bellekte 1 saat süre saklasın diye şöyle yaparız.
git config --global credential.helper "cache --timeout=3600"
Örnek
Şifrelerin nerede saklandığını görmek için şöyle yaparız
git config --system credential.helper
manager-core
Eğer manage-core yerine manager yazsaydı şifreler Windows'ta "Kimlik Bilgileri Yöneticisi" veya İngilizce olarak "Credentials Manager" içinde saklanırdı. Yani eskiden bilgiler Windows'ta saklanırdı.

Bilgileri görmek için şöyle yaparız
git credential fill
protocol=https
host=github.com
İki defa enter'a bas
Çıktı şöyle
protocol=https
host=github.com
username=...
password=...
-l seçeneği
Örnek
Ayarları listelemek için şöyle yaparız.
git config -l --show-origin
Örnek
Global ayarları listelemek için şöyle yaparız.
$ git config --global -l
veya şöyle yaparız.
$ git config -l
Çıktı olarak şunu alırız.
user.email=new@new.com
user.name=new
-f seçeneği
Örnek
Sadece submodule ile çalışıp doğru branch'e commit etmek için şöyle yaparız.
cd /path/to/your/parent/repo
git config -f .gitmodules submodule.<path>.branch <branch>




9 Aralık 2021 Perşembe

Prometheus

Giriş
Açıklaması şöyle
Prometheus is an open-source monitoring system that helps to periodically monitor the JVM Based systems.
Prometheus Sunucusu
Prometheus Sunucusu genellikle 9090 portunu dinleyecek şekilde çalıştırılıyor. prometheus.yml dosyasında sorgulanacak spring boot uygulamasını adresi ve ne kadar sıklıkla sorgulanacağı (poll) belirtiliyor.

Prometheus aldığı değerleri de bir time series veri tabanına kaydeder. Böylece veriler kaybolmaz. Veriyi görmek için Prometheus sunucusunun çalıştığı şu adrese gideriz. Ekran görüntüsü burada
http://localhost:9090/
Bir metric seçip "Execute" düğmesine tıklarsa karşımıza bir grafik çıkar

Metrics Adresi
Şu adrese gideriz
localhost:9090/metrics
Çıktısı şöyle
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.3545e-05
go_gc_duration_seconds{quantile="0.25"} 1.6924e-05
go_gc_duration_seconds{quantile="0.5"} 3.8626e-05
go_gc_duration_seconds{quantile="0.75"} 4.503e-05
go_gc_duration_seconds{quantile="1"} 0.000437923
...
Prometheus Exporter
Açıklaması şöyle
Prometheus is an exporter based monitoring tool that means on target machine a exporter is installed that takes the target machine health and then Prometheus server pulls that data from exporter. These data are called Metrics. The pulling of metrics happens over HTTP then Prometheus server scrapes the metrics from web.
Açıklaması şöyle
Exporters are standalone applications in the Prometheus ecosystem that exposes metrics for third-party systems that cannot be scraped easily.

Why do we need Exporters? Because some third-party systems have their internal states but Prometheus server cannot consume those data. To overcome this issue, Exporters are developed by a large number of Community contributors or organizations to export the internal state of third-party systems to a format that Prometheus can consume.
- Örneğin node_exporter işletim sistemini detaylarını Prometheus'a göndermek içindir. Hedef bilgisayarda çalışır

Service Discovery
Açıklaması şöyle
Service Discovery allows Prometheus to discover machines and/or services running somewhere. It can be anything that contains or returns a list of targets, such as a JSON file or an HTTP endpoint or as complex as Kubernetes API.

Why do we need service discovery? Because Prometheus reads all configuration from the configuration file at boot time, while the targets may come and go after Prometheus booted. To overcome this issue, Prometheus need to dynamically load this extra bit of information in runtime.

Graph Adresi
Şu adrese gideriz
http://localhost:9090/graph
Şeklen şöyle


Örnek
url : http://localhost:9090/graph
search by “process_cpu_usage”

Targets Adresi
Şu adrese gideriz. Scrape edilen hedefleri gösterir.
http://localhost:9090/targets
Örnek
Şeklen şöyle. Burada yaml dosyasında "job-name" alanında belirtilen spring uygulaması gösteriliyor.

Örnek
Şeklen şöyle. Burada yaml dosyasında "job-name" alanında belirtilen Jenkins uygulaması gösteriliyor.


prometheus komutu
Prometheus'u başlatmak için "./prometheus" komutu çalıştırılır

 -- config.file seçeneği
Şöyle yaparız
prometheus \
  --config.file=/etc/prometheus/prometheus.yml 
  --web.config.file=/etc/prometheus/web-config.yml
--version seçeneği
Şöyle yaparız
prometheus --version
prometheus, version 2.39.0 (branch: HEAD, revision: 6d7f26c46ff70286944991f95d791dff03174eea)
  build user:       root@bc053716806f
  build date:       20221005-05:09:43
  go version:       go1.19.1
  platform:         linux/amd64

Promtool komutu
Prometheus konfigürasyonunu doğrulamak için kullanılır.

PromQL
PromQL yazısına taşıdım

Pushgateway
Şeklen şöyle


Açıklaması şöyle
It uses a feature called a Pushgateway that lets you push time series metrics data to it in cases where data cannot be scraped or pulled.
Açıklaması şöyle
Pushgateway is a standalone application in the Prometheus ecosystem that exposes metrics for ephemeral and batch jobs that can’t be scraped.

We do we need Pushgateway? Because the lifecycle of those jobs are too short for Prometheus continuously scraping metrics. Instead, those jobs push metrics to pushgateway and then Prometheus scrapes metrics from pushgateway. Note that Pushgateway does not change the “pull” model of Prometheus server.

web-config.yml
Basic authentication için şöyle yaparız. Şifre bcrypt ile şifrelidir
basic_auth_users:  
  {your_desired_username}:{your_encrypted_password}

prometheus.yml
prometheus.yml yazısına taşıdım

alert_rules.yml
Bu dosyada alarm durumu tanımlı. Belli bir eşik aşılınca alert gönderilir. Şeklen şöyle
docker/
├─ alert-manager/
│ ├─ config/
│ │ ├─ alertmanager.yml
├─ grafana/
│ ├─ grafana.ini
├─ prometheus/
│ ├─ config/
│ │ ├─ alert-rules.yml
│ │ ├─ prometheus.yml
docker-compose.yaml
prometheus.yml dosyasında alert-rules dosyası şöyle belirtilir
rule_files:
  - 'alert-rules.yml'
alerting:
  alertmanagers:
    - scheme: http
    - static_configs:
        - targets: ['host.docker.internal:9093']
Örnek
alert-rules.yml şöyledir
groups:
  - name: tutorial-rules
    rules:
      # Triggers a critical alert if a server is down for more than 1 minute.
      - alert: ServerDown
        expr: up < 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."
Açıklaması şöyle
Inside the alert-rules.yml file, under the groups section, we've created a named group called tutorial-rules. This group holds the instructions for our rules. Here’s a breakdown of what’s happening with our example rule:

alert: ServerDown: This line assigns a name to our alert, 'ServerDown'. It's like a label for the alert, helping us identify what's going on.
expr: up < 1: This line is the heart of the rule. It defines the condition for the alert. In simple terms, it checks if the 'up' metric (which tells us if a server is running) is less than 1. If it's less than 1, it means the server is down. You can find more examples of query basics here.
for: 1m: This part tells Prometheus to wait for 1 minute before triggering the alert. It gives a little time, just in case the server goes down briefly.
labels:: These are like tags we attach to the alert. In our case, we're labeling the severity as 'critical'.
annotations:: These are additional details we add to the alert. The summary and description lines help us understand the situation better. They tell us which server and job are affected and how long the server has been down.

Örnek
Bir örnek burada

Docker
Prometheus Docker yazısına taşıdım

Compose Dosyası
Docker Compose yazısına taşıdım

AlertManager
Açıklaması şöyle
Prometheus doesn’t require any agents or applications to be installed on your server fleet in order to collect data, and it leverages a component called AlertManager, which manages alerts and sends notifications via email, on-call systems, and/or group collaboration tools like Slack.
Açıklaması şöyle
Prometheus server as a client application runs alert rules periodically and push alerts to Alertmanager. Alertmanager then takes care of deduplicating, grouping, and routing them to the correct receiver integrations.

Why do we need Alertmanager? There is no clear answer though but I think it is due to trade-off. We could have built the alerting functionality into Prometheus server but that have a drawback. Due to the distributed nature of Prometheus server, alerts coming from different Prometheus servers could be fired even they have very similar information. Features like deduplicating needs some form of consensus. Introducing consensus algorithm implementation into Prometheus server significantly increases the complexity and maintenance overhead. Instead, the decision was to introduce a central component “Alertmanager” to deal with these features.
AlertManager kendi başına bir uygulama. Eğer alert olursa bunu e-posta, Slack gibi bir kanala göndermek içindir. Bir örnek burada

AlertManager uygulamasının çıktısını görmek için şu adrese gideriz.
http://localhost:9093/#/alerts
alertmanager.yaml Dosyası
alert-rules.yaml ile belirtilen alert durumunun nasıl gönderileceği bu dosyada tanımlı

Örnek - Telegram + E-Posta
Şöyle yaparız
route:
  receiver: tutorial-alert-manager
  repeat_interval: 1m
receivers:
  - name: 'tutorial-alert-manager'
    telegram_configs:
      - bot_token: tutorial_token
        api_url: https://api.telegram.org
        chat_id: -12345678
        parse_mode: ''
    email_configs:
      - to: 'tutorial.inbox@gmail.com'
        from: 'tutorial.outbox@gmail.com'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'username'
        auth_password: 'password'
Açıklaması şöyle
Here, receiver points to the name of the receiver you're setting up. The repeat_interval specifies how often to send repeat notifications, ensuring crucial alerts aren't missed. Our example illustrates the use of two distinct receivers.

  • The first receiver is configured to send alerts via Telegram. It uses the provided bot token, chat ID, and Telegram API.
  • The second receiver is set up for email notifications. In this configuration, we specify an email receiver that sends notifications to a specified email address. You will need to replace the placeholder values (tutorial.inbox@gmail.com, tutorial.outbox@gmail.com, smtp.gmail.com:587, username, password) with your own email and SMTP server details.