Cloud Computing - Infrastructure as a Service (IaaS)

Giriş

Açıklaması şöyle

IaaS (Infrastructure as a Service) gives you networking, storage, and VMs. In IaaS, your VMs will be maintained fully by your internal staff, however, there are add-ons that can automate services like backups and even patching in some cases. You can compare this to using a hosting provider who provides the hardware, and you install, configure, and maintain anything housed on that VM.

Bana Windows 7 çalıştıran, i3 işlemci gücünde 1000 tane PC lazım, isteğinin yerine getirilmesi bu seviyede olur.

Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) gibi bulut sağlayıcıları IaaS hizmeti verirler

Cloud Computing - Software as a Service (SaaS)

Giriş

Açıklaması şöyle

- SaaS delivers software applications over the internet, eliminating the need for users to install, maintain, or manage the software on their devices.

- SaaS providers host and maintain the software, making it accessible to users through web browsers or dedicated applications.

- Common examples of SaaS applications include Gmail, Salesforce, and Microsoft 365.

Açıklaması şöyle.

In a SaaS you may only pay for the users you have registered. but not every user uses all the features you are paying for. In an IaaS you may pay for servers of a predefined capacity, regardless of the really used computing power, i.e. you pay for idle time as well.

Örnek - User Management as SaaS

Okta, Auth0, Frontegg gibi sağlayıcılar var

Örnek - Log Ingestion as SaaS

Bir örnek şöyle

For example, you don't need to configure Logstash and ElasticSearch replication. There are at least half a dozen companies that will ingest all the logs from your application, allow you to search them, and delete them after 90 days so you don't pick up a GDPR fine in the future. Many companies will also deliver email and SMS with nothing more than a credit card on file and an API client. Twenty years ago, those grumpy systems administrators would have been configuring Sendmail, reading logs, drinking coffee, and cursing — all at the same time.

SaaS üreticileri açısından Customer Acquisition Cost (CAC) önemli olabilir. Açıklaması şöyle

Your CAC is how much it costs you to acquire a new customer. This metric is measured over a specific timeframe, and a variety of factors should be considered – from advertising and sales costs to salaries and more. The CAC is also influenced by your industry and your business’s customer acquisition strategies.

The basic formula for calculating CAC for SaaS is as follows:

Total costs of sales and marketing / Total number of acquired customers = CAC

Cloud Computing - Function as a Service - (FaaS) - Serverless Computing

Giriş

Event'lere bağlanabilen fonksiyonlar yazarak, servisleri çağırabilmemize sağlar. AWS Lambda, Azure Functions gibi örnekler verilebilir

GitHub SSH Key

Giriş

1. Kendi bilgisayarımda public/private key dosyaları oluştururum. IntelliJ terminal penceresinden şöyle yaparız. Burada -C ile e-posta adresi belirtiliyor, şart değil. Ben yapmadım

ssh-keygen -t ed25519 -C "your_email@example.com"

Windows kullanıyorsak dosyalar şurada

C:\Users\acelya\.ssh

Eğer Git Bash ile 

ls -al ~/.ssh

yapsak bile yine aynı dizini gösteriyor. 

2. id_ed25519.pub dosyasındaki içindeki satırları kopyalayıp, GitHub'da https://github.com/settings/keys adresinden yeni bir SSH key oluştururum

3. .git/config dosyasında depoya erişmek için kullanılan yöntem var. 

- Eğer depo ismi git@ ile başlıyorsa SSH kullanılır

- Eğer depo ismi https  ile başlıyorsa HTTPS kullanılır

Apache Arrow - Columnar Memory Format

Giriş

Açıklaması şöyle

Apache Arrow is a cross-language development framework for in-memory data. It provides a standardized columnar memory format for efficient data sharing and fast analytics. Arrow employs a language-agnostic approach, designed to eliminate the need for data serialization and deserialization, improving the performance and interoperability between complex data processes and systems.

Maven

Şu satırı dahil ederiz

<dependency>
  <groupId>org.apache.arrow</groupId>
  <artifactId>arrow-memory</artifactId>
  <version>6.0.1</version>
</dependency>

<dependency>
  <groupId>org.apache.arrow</groupId>
  <artifactId>arrow-vector</artifactId>
  <version>6.0.1</version>
</dependency>

Örnek

Yazma için şöyle yaparız

import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.*;
import org.apache.arrow.vector.ipc.*;
import org.apache.arrow.vector.util.*;

// Set up the allocator and the schema for the vector
try (RootAllocator allocator = new RootAllocator(Integer.MAX_VALUE);
  VarCharVector vector = new VarCharVector("vector", allocator);
  ArrowWriter writer = new ArrowWriter(vector, new Schema(Collections.
    singletonList(vector.getField())))) {

  // Write data to the vector
  vector.setSafe(0, "Apache".getBytes());
  vector.setSafe(1, "Arrow".getBytes());
  vector.setSafe(2, "Java".getBytes());
  vector.setValueCount(3);

  // Write vector to a file
  try (FileOutputStream out = new FileOutputStream("arrow-data.arrow")) {
    writer.writeArrow(out.getChannel());
  }
}

Okuma için şöyle yaparız

// Now, let's read the data we just wrote
try (RootAllocator allocator = new RootAllocator(Integer.MAX_VALUE);
  ArrowReader reader = new ArrowReader(new FileInputStream("arrow-data.arrow")
    .getChannel(), allocator)) {

  // Read schema and load the data
  reader.loadNextBatch();

  // Get the vector
  try (VarCharVector vector = (VarCharVector) reader.getVectorSchemaRoot()
    .getVector("vector")) {
    // Iterate over the values in the vector
    for (int i = 0; i < vector.getValueCount(); i++) {
      System.out.println(new String(vector.get(i)));
    }
  }
}

Apache Flink SQL

Giriş

Bir Kafka Topic'i sorgulayabilir

Not : HTTP Connector For Flink SQL yazısına bakabilirsiniz

Örnek

Şöyle yaparız

CREATE TABLE `ssb`.`Meetups`.`openskyairport` (
  `icao24` VARCHAR(2147483647),
  `firstSeen` BIGINT,
  `estDepartureAirport` VARCHAR(2147483647),
  `lastSeen` BIGINT,
  `estArrivalAirport` VARCHAR(2147483647),
  `callsign` VARCHAR(2147483647),
  `estDepartureAirportHorizDistance` BIGINT,
  `estDepartureAirportVertDistance` BIGINT,
  `estArrivalAirportHorizDistance` VARCHAR(2147483647),
  `estArrivalAirportVertDistance` VARCHAR(2147483647),
  `departureAirportCandidatesCount` BIGINT,
  `arrivalAirportCandidatesCount` BIGINT,
  `ts` VARCHAR(2147483647),
  `uuid` VARCHAR(2147483647),
  `eventTimeStamp` TIMESTAMP(3) WITH LOCAL TIME ZONE METADATA FROM 'timestamp',
  WATERMARK FOR `eventTimeStamp` AS `eventTimeStamp` - INTERVAL '3' SECOND
) WITH (
  'scan.startup.mode' = 'group-offsets',
  'deserialization.failure.policy' = 'ignore_and_log',
  'properties.request.timeout.ms' = '120000',
  'properties.auto.offset.reset' = 'earliest',
  'format' = 'json',
  'properties.bootstrap.servers' = 'kafka:9092',
  'connector' = 'kafka',
  'properties.transaction.timeout.ms' = '900000',
  'topic' = 'openskyairport',
  'properties.group.id' = 'openskyairportflrdrgrp'
)

Şöyle yaparız

select * from openskyairport

Apache NiFi

Giriş

İlk olarak burada gördüm. Bir REST çağrısı yapıp sonucu Kafka'ya yazabiliyor

Paxos Consensus Algorithm - Lidersiz Çalışır

Giriş

Açıklaması şöyle

Paxos achieves consensus in a three-phased, leaderless, majority-wins protocol. While Paxos is effective in driving correctness, it is notoriously difficult to understand, implement, and reason about. This is partly because it obscures many of the challenges in reaching consensus (such as leader election, and reconfiguration), making it difficult to decompose into subproblems.

Apache Flink Modları

Giriş

Açıklaması şöyle

Flink can execute application in one of these three ways:

1. Application Mode

2. Per-Job Mode

3. Session Mode

The main differences between these three ways are:

- Cluster lifecycle and resource isolation

- Whether the application’s main() method is executed on the client or on the cluster

Şeklen şöyle

1. Application Mode

Yeni bir cluster yaratır. Açıklaması şöyle. İşler birbirlerinden tam anlamıyla yalıtılır

In this mode, you need to deploy the Flink image everytime having your job placed in the lib folder. What does this means?

Basically, the flink job is a fat jar containing your logic + dependencies. Now the flink cluster in application mode, running your job is especially created to execute your job only. That means no other job shares your flink cluster. So let’s say you may have 3 jobs to run, so in application mode you need to create 3 flink clusters.

2. Session Mode

Açıklaması şöyle

In this mode, you needn’t have to deploy the whole flink cluster again. All required is to to deploy the fat jar with your fixes/enhancements & deploy it.

Now how you deploy is to stop the current jar execution & start the new one. Rest api’s are provided to do this as well command prompt based shell script can do the magic as well to stop & submit the job.

Amazon Web Service (AWS) eksctl komutu

Giriş

EKS ile çalışmayı kolaylaştırır. Açıklaması şöyle

We need to install the eksctl command-line utility to manage the EKS cluster and the Kubernetes command-line tool kubectl.

create cluster seçeneği

Örnek

Şöyle yaparız

> eksctl create cluster \ 
--name microservices \ 
--region eu-central-1 \ 
--node-type t2.small \ 
--nodes 2

Açıklaması şöyle

It will make a cluster with two worker nodes of type “t2.small” in the region “eu-central-1” with the name “microservice.”

In the background, eksctl uses CloudFormation to create the cluster, which usually takes 10–15 minutes. 

Açıklaması şöyle

From the output, it is evident that it has created two nodes and one node group. Also, it has saved the kubectl config file in ./.kube/config. In case you already have minikube or microk8s, you have to mention the ./.kube/config file as the kubeconfig parameter in the command.

Creating an EKS cluster will take around 15 minutes. Once the cluster is ready, you can check it with the following command:

> kubectl get nodes --kubeconfig ~/.kube/config

Örnek

Şöyle yaparız

$ eksctl create cluster \
  --name my-cluster \
  --version 1.18 \
  --region us-west-2 \
  --nodegroup-name my-nodes \
  --node-type t3.medium \
  --nodes 3

Örnek

Şöyle yaparız

$ eksctl create cluster \
 --name Democluster\
 --version <1.21> \
 --without-nodegroup

get cluster seçeneği

Örnek

Şöyle yaparız

$ eksctl get cluster

Logstash

logstash.conf Dosyası

Örnek - filebeat'ten okur

Şöyle yaparız

input {
  beats {
    port => "4561"
    host => "0.0.0.0"
  }
}

filter { }

output {
  elasticsearch {
    hosts => ["0.0.0.0:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Örnek - stdout + elasticsearch 

Şöyle yaparız

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  file {
    path => "<Log File Full Path>"
    start_position => "beginning"
  }
}

output {
 stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["https://localhost:9200"]
    ssl_certificate_verification => false
    ssl => true
    index => "elkdemoindex"
    user => "elastic"
    password => "<Elastic Search Password>"
  }
}

Örnek - TCP'den okur

Şöyle yaparız

input {
    tcp {
        port => 4560
        codec => json_lines
    }
}

filter { }

output {
    
    elasticsearch {
        hosts  => ["http://elasticsearch:9200"]
        index  => "operationlog-%{app_name}-%{+YYYY-MM-dd}"
    }
    
}

AWS Systems Manager

Giriş

Açıklaması şöyle

With Systems Manager, users exchange their AWS credentials for temporary shell access ....

EC2 instance'ları için AWS Resource Groups içinde tag yaratarak AWS Systems Manager ile hepsine shell komutunu çalıştıran bir örnek burada

AWS Systems Manager Parameter Store

Parameter Store ayarlarına sol taraftaki menüden erişilir. Açıklaması şöyle

AWS Systems Manager Parameter Store provides secure storage and management of secrets. It allows storing of data such as passwords, database strings, Amazon Elastic Compute Cloud (Amazon EC2) instance IDs and Amazon Machine Image (AMI) IDs, and license codes as parameter values. It can store values as plain text or encrypted data.

AWS Systems Manager Parameter Store vs AWS Secrets Manager

AWS Secrets Manager daha fazla özellik sağlıyor. Örneğin secret rotation gibi.

UML Package Diagram

Giriş

Açıklaması şöyle

The UML package diagram describes the structures of the designed system at the level of packages or the dependencies between the packages that make up a model.

Paketler Arası Bağımlılıklar

Bağımlılıkları gösteren bir örnek burada. Bağımlılıklar kesikli açık ok (dashed open arrow) ile gösterilir. Bağımlılıklar için stereotype etiketler kullanılabilir. Bunların bazıları şöyle

1. <<import>>

Açıklaması şöyle

One package imports the functionality of another package. Only public elements in the package will be imported.

2. <<access>>

Açıklaması şöyle

One package requires assistance from functions of another package. 

3. <<merge>>

Açıklaması şöyle. Bence extends diye adlandırılsaydı daha iyi olurdu

It is the directed relationship between 2 packages that indicates that the content of one package is extended by the contents of another package. 

AWS Transcribe

Giriş

Açıklaması şöyle

Amazon Transcribe is a service that utilizes Machine Learning models to convert speech to text automatically. It offers various features that can enhance the accuracy of the transcribed text, such as language customization, content filtering, multi-channel audio analysis, and individual speaker speech partitioning. Amazon Transcribe can be used as a standalone transcription service or to add speech-to-text capabilities to any application. You can transcribe media in real-time (streaming) or you can transcribe media files located in an Amazon S3 bucket (batch).

aws ecr seçeneği - Container Registry

create-repository seçeneği

--repository-name ile ismi belirtilen yeni bir repository yaratılır

Örnek

Açıklaması şöyle

The container should be published to a registry. For this end, we’ll use Amazon ECR, a managed container registry to store, share, and deploy containers in the AWS Cloud.

...

Now, for each microservice container image, we need to create an ECR repository. Please note the repository name should exactly match the container image repository name.

Here’s the command to create a repository in ECR for the Order container image:

Şöyle yaparız

> aws ecr create-repository --repository-name microservice-customer

Çıktısı şöyle

{
  "repository": {
  "repositoryArn": "arn:aws:ecr:eu-central-1:877546708265:repository/microservice-customer",
  "registryId": "877546708265",
  "repositoryName": "microservice-customer",
  "repositoryUri": "877546708265.dkr.ecr.eu-central-1.amazonaws.com/microservice-customer",
  "createdAt": "2021-03-04T00:18:33+01:00",
  "imageTagMutability": "MUTABLE",
  "imageScanningConfiguration": {
    "scanOnPush": false
  },
  "encryptionConfiguration": {
    "encryptionType": "AES256"
  }
}

Yerel image yeniden tag'lenir ve push edilir. Şöyle yaparız

> docker tag 652da8e2130b 877546708265.dkr.ecr.eu-central-1.amazonaws.com/microservice-customer:1.0.0

> aws ecr get-login-password \
  --region eu-central-1 | \
  docker login \
    --username AWS \
    --password-stdin \
    877546708265.dkr.ecr.eu-central-1.amazonaws.com

> docker push 877546708265.dkr.ecr.eu-central-1.amazonaws.com/microservice-customer:1.0.0

delete-repository seçeneği

--repository-name ile ismi belirtilen repository silinir

Örnek

Şöyle yaparız

aws ecr delete-repository --repository-name hello-app-runner --force

get-login-password seçeneği

docker veya podman ile login olup image push etmek için şifre gerekir. Şifreyi ortamdan çekmek için bu seçenek kullanılır

Örnek - docker

Login olmak için şöyle yaparız

aws ecr get-login-password --region eu-central-1 | 
docker login 
  --username AWS 
  --password-stdin 378612673110.dkr.ecr.eu-central-1.amazonaws.com/java-demo

Örnek - podman

Şöyle yaparız

# Create an ECR repository
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_REGION=us-east-1
aws ecr create-repository --repository-name hello-app-runner

# Authenticate with an external container registry and push the generated image
podman login --username AWS --password $(aws ecr get-login-password \
     --region ${AWS_REGION}) ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
podman tag hello-app-runner:latest \
  ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/hello-app-runner:latest

H.264 Nedir? - Bir Codec Standardı

Giriş

H.264 bir codec standardı. Açıklaması şöyle. Yani video cihazından görüntüyü yakalar ve dijital hale getirir.

H.264 is one of the most popular codecs for live video streaming, and it can be used to create video files in various resolutions up to 8k.

Bu standardı gerçekleştiren farklı encoder'lar var. Farkı vurgulayan bir açıklama şöyle

Consider the case for H.264. The standard's name is H.264 – that's not the name of the actual encoder. Mainconcept is a very good commercial encoder, whereas x264 is a free and open source one. 

...

The mere fact that you can optimize encoding makes for a competition here. Both encoders will deliver a standardized bitstream that can always be decoded by a H.264-compliant decoder.

Sıkıştırma

Açıklaması şöyle. Sıkıştırma yöntemlerinden bir tanesi interframe compression. İki frame arasındaki fark gönderilir.

How does H.264 encoding work?

H.264 is a video compression standard that is commonly used to encode video files for a variety of purposes, including streaming over the internet. H.264 uses a combination of techniques to reduce the size of video files without significantly impacting the quality of the video. One of the key techniques used by H.264 is known as “interframe compression.” This technique involves analyzing the differences between consecutive frames of the video and only storing the information that has changed from one frame to the next. By storing only the differences between frames, H.264 is able to significantly reduce the amount of data that is required to represent the video. H.264 also uses techniques such as quantization, entropy coding, and motion compensation to further reduce the size of the video data. These techniques involve analyzing the video data and removing redundant or irrelevant information, as well as using advanced mathematical algorithms to represent the data more efficiently.