Quorum Nedir ? - Çoğunluğu Sağlayan Sayı

Giriş

Not : Yazılım Mimarisi - Replica yazısına bakabilirsiniz.

Açıklaması şöyle. Quorum oy çokluğu demek

Distributed consensus based on a quorum model

How much is Enough?

Açıklaması şöyle. Yani çoğunluğu sağlayan en küçük sayı Quorum olarak yeterli.

We defined Quorum as the minimum number of servers that need to acknowledge an operation before it can be considered successful. But what's a good number, such that we get both, good application performance & consistency.

We generally prefer for majority of nodes in the cluster to acknowledge any operation for it to be considered successful. Thus, for a N node cluster, Quorum should be of N/2 + 1 nodes.

What if I choose Quorum to be > N/2 + 1? 

Well, you will have more number of nodes to acknowledge your changes, and so you'll have a performance hit, when compared to choosing N/2 + 1 as your Quorum. Refer to Scenario 1 for understanding the impact.

What if I choose Quorum to be < N/2 + 1? 

In this case, only minority of the nodes in the cluster are guaranteed to have the changes. In case those nodes go down/are network partitioned, the changes wouldn't be visible to the end users & would have consistency issues. Refer to Scenario 2 for understanding the impact.

Bir başka açıklama şöyle. 

The basis of the algorithm is that for any action to take place there must be quorum which is decided by (N/2) +1, with N voting members of the distributed system. When an action is requested, it must be voted on by all voting members. If it receives greater than 50% of the votes, then the action can take place. In this case, the action is written to a log on each node which is the source of truth. By having distributed consensus, you get the security of knowing anything written to the log is an allowed action as well as having log replication and leader elections.

Real Life Usage

Açıklaması şöyle. 

1) Consensus algorithms like Paxos, Raft are all quorum based.

2) Cassandra uses write quorum to ensure data consistency, where a write is considered successful only after its replicated to at least quorum of replica nodes.

3) Leader Election happens only if leader gets vote from majority of the servers i.e Quorum.

Docker ve Kafka Connect

Örnek

Şöyle yaparız. jars dizini altında ojdbc8.jar dosyası var.

FROM confluentinc/cp-kafka-connect:5.3.0

ENV KAFKA_HEAP_OPTS "-Xms1G -Xmx3G"
EXPOSE 8083 8083

ADD jars/* /etc/kafka-connect/jars/

Apache Beam - Batch and Streaming Data Processing

Giriş

Şeklen şöyle. Apache Beam farklı dillerde kodlanabilir ve farklı Runner'lar kullanabilir

Gradle

Şöyle yaparız

implementation("org.apache.beam:beam-sdks-java-core:2.45.0")
runtimeOnly("org.apache.beam:beam-runners-direct-java:2.45.0")

Örnek

Şöyle yaparız

public class App {
  public static void main(String[] args) {
    PipelineOptions options = PipelineOptionsFactory.create();
    // Create pipeline
    Pipeline p = Pipeline.create(options);
    // Read text data from Sample.txt
    PCollection<String> textData = p.apply(TextIO.read().from("Sample.txt"));
    // Write to the output file with wordcounts as a prefix
    textData.apply(TextIO.write().to("wordcounts"));
    // Run the pipeline
    p.run().waitUntilFinish();
  }
}

Çıktı wordcounts-00000-of-00001 dosyasındadır. Açıklaması şöyle

1. Create a PipelineOption.
2. Create a Pipeline with the option.
3. Add the logic to read data from Sample.txt to the pipeline and get the return value as PCollection, which is an abstraction of dataset in Apache Beam.
4. Add another step to write the return value in the previous step to output file with name starting with wordcounts.
5. Lastly, run and finish the pipeline.

Prometheus Blackbox Exporter

Giriş

Açıklaması şöyle

The blackbox-exporter is an exporter that can monitor various endpoints — URLs on the Internet, your LoadBalancers in AWS, or Services in a Kubernetes cluster, such as MySQL or PostgreSQL databases.

Blackbox Exporter can give you HTTP response time statistics, response codes, information on SSL certificates, etc.

Açıklaması şöyle

The Prometheus Blackbox Exporter is an open-source tool developed under the Prometheus umbrella. Once installed, it opens an HTTP port 9115 and exposes two metric paths

1. The /metrics endpoint returns metrics about the Blackbox Exporter running
2. The /probe endpoint retrieves metrics from a target supplied as a request parameter.

The latter is the more interesting endpoint, allowing us to check multiple targets. Targets can be HTTP endpoints, but ICMP (Ping), DNS, or raw TCP is also permitted (see the documentation for details).

probe Endpoint

Açıklaması şöyle

The /probe endpoint accepts the following parameters :

target - the HTTP endpoint to check (e.g. https://ping7.io)

module - the response check we want to conduct. OOTB, the Blackbox exporter comes with the http_2xx and the http_post_2xx module. Both check for a 2xx HTTP response code and query the target via HTTP GET or POST request.

debug - can be set to true to retrieve debug information for the current probe.

You can verify your Blackbox Exporter installation by calling it via 

curl:curl "http://localhost:9115/probe?target=https://ping7.io&module=http_2xx"

relabel_configs 

Açıklaması şöyle

Using the magic of relabel_configs in Prometheus, you can query multiple search terms in a single scrape job definition.

Örnek

Şöyle yaparız

scrape_configs:
- job_name: 'blackbox'
    metrics_path: /blackbox/probe
    scheme: https
    authorization:
      # Your ping7.io API token stored in this file
      credentials_file: ping7io-api-token
    params:
      module: [http_2xx]
      target: ["https://www.zalando.de/katalog/"]
    static_configs:
      - targets:
        - schuhe
        - hose
    - schwarz
    - sale
    relabel_configs:
      # store target as search_term
      - source_labels: [__address__]
        regex: '(.*)'
        target_label: search_term

      # build new target by concatenating
      # target param and static target config
      - source_labels: [__param_target, __address__]
        separator: "?q="
        target_label: __param_target

      # store new target as instance
      - source_labels: [__param_target]
        target_label: instance

      # use the ping7.io exporter endpoint
      - target_label: __address__
        replacement: check.ping7.io

Açıklaması şöyle

... you list the search terms to query as targets. In the relabel_configs we append them to the target parameter and query the Blackbox Exporter for response time metrics for this specific search term.

In the example below, we query https://www.zalando.de/katalog/?q=schuhe as a first target for response times. After relabeling, the Prometheus is scraping the URL https://check.ping7.io/blackbox/probe?target=https://www.zalando.de/katalog/?q=schuhe&module=http_200 for metrics. And the best thing is: you can add as many search terms as you like.

Kafka OAuth

Giriş

Açıklaması şöyle

KIP-255 introduced OAuth Authentication via SASL/OAUTHBEARER, and KIP-768 added default implementation for Login/Validator Callback Handlers(aka OIDC) that use client_credentials Oauth2.0 grant type.

Burada hem broker hem de client tarafında OAuth kullanmak için bir örnek var

PASETO - Platform-Agnostic Security Token

Giriş

Açıklaması şöyle

PASETO (Platform-Agnostic SEcurity TOken) is a specification and reference implementation for secure stateless tokens. It is pronounced paw-set-oh (pɔːsɛtəʊ).

PASETO encodes claims to be transmitted in a JSON (RFC8259) object and is either encrypted symmetrically or signed using public-key cryptography.

JPaseto için bir yazı burada

PASETO Vs JOSE (JWS, JWE and JWT)

Açıklaması şöyle. Yani sadece tanımlı olan şifreleme algoritması kullanılabilir

The key difference between PASETO and the JOSE family of standards (JWS [RFC7516], JWE [RFC7517], JWK [RFC7518], JWA [RFC7518], and JWT [RFC7519]) is that JOSE allows implementors and users to mix and match their own choice of cryptographic algorithms (specified by the “alg” header in JWT), while PASETO has clearly defined protocol versions to prevent unsafe configurations from being selected.

PASETO token format

Şöyledir

version.purpose.payload

veya

version.purpose.payload.footer

Redis - Key-Value Veri Yapısı

SET

Şöyle yaparız

SET lock_name arbitrary_lock_value NX EX 10 # acquire the lock
# ... do something to the shared resource
DEL lock_name # release the lock

SET NX

Atomic olarak çalışır. Eski hali SETNX. Sonradan SET komutuna NX parametresi haline geldi. Açıklaması şöyle

Because Redis has atomic writes, we can use a SET NX operation to only set the key if it’s not there. This means that Redis guarantees that only one worker will the race for a particular key. The official documentation says that the Redlock algorithm is preferred over SET NX because it “but offers better guarantees and is fault tolerant.” We are not looking to use a lock with Redis, however, just a one-way gate that prevents multiple execution. As long as Redis can prevent the race condition where two workers read the same value for the gate and both pass through it, that is all we need.

Örnek

Şöyle yaparız

SETNX lock_name true

DEL lock_name

EXPIRE

Örnek

Şöyle yaparız

SETNX lock_name arbitrary_lock_value
EXPIRE lock_name 10