Helm ve Prometheus

Örnek

Şöyle yaparız

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack

Örnek

Şöyle yaparız

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ kubectl create ns monitoring
$ helm -n monitoring install prometheus prometheus-community/kube-prometheus-stack

Exporter

Her exporter bir de servis yaratır. Bu servisin metrics adresine gidersek exporter hakkında bilgiler görürüz.

127.0.0.1:9108/metrics

Örnek - blackbox-exporter

Blackbox Exporter yazısına bakabilirsiniz. Şöyle yaparız

helm -n monitoring upgrade 
  — install prometheus-blackbox prometheus-community/prometheus-blackbox-exporter

Örnek - elasticsearch-exporter

Şöyle yaparız

helm install prometheus-elasticsearch-exporter \ 
  prometheus-community/prometheus-elasticsearch-exporter

Google Cloud Firestore - NoSQL Document Database

Cloud Datastore and Firestore

Açıklaması şöyle

Datastore is a highly scalable NoSQL Document Database. It automatically scales and partitions data as it grows. Recommended for use cases needing flexible schema with transactions. Examples: User Profile and Product Catalogs. Datastore can handle upto a few terabytes of data.

Here are some of the important features:

1. Supports Transactions, Indexes and SQL-like queries (GQL)

2. Does NOT support Joins or Aggregate (sum or count) operations

Firestore is the new version of Datastore. I call it Datastore++. It is optimized for multi-device access. It provides an offline mode and data synchronization across multiple devices - mobile, IoT etc.

Choosing Between Cloud Firestore, Datastore vs Cloud BigTable

Açıklaması şöyle

Cloud Datastore is managed serverless NoSQL document database. It provides ACID transactions, SQL-like queries, and indexes. It is designed for transactional mobile and web applications.

Firestore is the next version of Datastore with additional capabilities like Strong consistency and Mobile and Web client libraries.

Firestore and Datastore are recommended for small to medium databases (0 to a few Terabytes).

Cloud BigTable on the other hand, is a managed, scalable NoSQL-wide column database. It is NOT serverless (You need to create instances).

BigTable is recommended for data sizes greater than 10 terabytes. It is usually used for large analytical and operational workloads.

BigTable is NOT recommended for transactional workloads. It does NOT support multi-row transactions - it supports ONLY single-row transactions.

Google Cloud Bigtable - Wide Column NoSQL Veri Tabanı

Giriş

Açıklaması şöyle

Bigtable is a distributed key/value storage system built at Google in 2005. It is designed to scale to store billions of rows and handle millions of reads/writes per second.

Bir başka açıklama şöyle. Yani sadece satır için transaction desteği verir

BigTable is recommended for data sizes greater than 10 terabytes. It is usually used for large analytical and operational workloads.

BigTable is NOT recommended for transactional workloads. It does NOT support multi-row transactions - it supports ONLY single-row transactions.

Bir başka önemli nokta şöyle

It is NOT serverless (You need to create instances).

Nerede Kullanılır

Açıklaması şöyle

A classic use case for Bigtable is storing sensor data and then running MapReduce jobs over it. This type of use case is a perfect fit for Bigtable because it typically requires high throughput and low latency. Additionally the nature of this data is likely parse with high cardinality.

Consistency Primitives

Açıklaması şöyle

Guarantees: Bigtable supports strong consistency within a single region and supports eventual consistency in cross regional deployments. Strong consistency means that all readers and writers get the same consistent view of the data.

Transactions: Bigtable does not support general purpose transactions. However, it does support single row transactions. A single row transaction enables reading and updating a single row as an atomic operation.

Intra-Row Guarantees: All updates to a single row are atomic. This is impressive and useful given the high cardinality of columns within a row.

Kullanımıyla ilgili bir açıklama şöyle

- A table is lexicographically sorted by row key. This enables schema designers to control the relative locality of their data by carefully selecting row key.

- A single table is designed to have on the order of 100 column families. Within each column family an arbitrary number of qualifiers can be used.

- Bigtable is great at modeling sparse data because if a column qualifier is not specified it does not take up any space in the row. Therefore a typical use case of Bigtable will involve having million of unique qualifiers within a table but each individual row will be smallish because it will be sparse relative to the set of all column qualifiers in the table.

- All data is immutable in Bigtable. When a new record is written either a new qualifier is added to a family or a new timestamp is added to a cell — data is never modified.

- The timestamps in the cells can either be assigned by the user or assigned by Bigtable. If the user assigns timestamps it is the responsibility of the user to ensure the timestamps are unique.

- All data in Bigtable (with one small exception we will ignore) are simply strings.

BigTable içinde bir sürü tablo olabilir. Ancak row mantığı biraz farklı. Şeklen şöyle

Açıklaması şöyle

Row Key

As part of a table’s schema a row key must be defined. A row key uniquely identifies a single row within a table. In our example employeeID was selected as the row key and we are looking at the row where employeeID=25.

Column Family - Aslında Map'in İsmi

As part of a table’s schema column families must be defined. Column families are used to store buckets of related entities. Our example shows two column families Contact Information and Manager Rating.

Column Qualifier - Map İçindeki Key Değerleri, Value Cell'dir

Within a family there can be arbitrary qualifiers. The qualifiers within a family should be related to each other. Qualifiers should be thought of as data rather than as part of the schema.

Cell and Timestamped Value - Cell İçinde Map Vardır. Map'in Key Değeri Timestamp

A row key, column family and column qualifier uniquely identify a single cell. A cell holds a collection of values. These values are organized into a map where the key is a timestamp and the value is a piece of user data.

Google Cloud SQL - İlişkisel Veri Tabanı

Cloud SQL Nedir?

Açıklaması şöyle

Cloud SQL is a Fully Managed Relational Database service.

Here are some of the important features:

1. Supports MySQL, PostgreSQL, and SQL Server

2. Regional Service providing High Availability (99.95%)

3. Option to use SSDs or HDDs (For best performance: use SSDs)

4. Automatic encryption (tables/backups), maintenance and updates

5. High availability and failover: Create a Standby with automatic failover

6. Read replicas for reading workloads - Options: Cross-zone, Cross-region and External (NON-Cloud SQL DB)

7. Automatic storage increase without downtime (for newer versions)

8. Point-in-time recovery: Enable binary logging

9. Backups (Automated and on-demand backups)

Cloud SQL vs Cloud Spanner

Cloud Spanner yazısına bakabilirsiniz. Açıklaması şöyle.

Use Cloud Spanner(Expensive) instead of Cloud SQL for relational transactional applications if:

1. You have huge volumes of relational data (TBs) OR

2. You need infinite scaling for a growing application (to TBs) OR

3. Do you need a Global (distributed across multiple regions) Database OR

4. You need higher availability (99.999%)

aws configure seçeneği

Giriş

Yaratılan "Access Key" değerlerini içeri alır. Bu komutun sonucunda 

1. ~/.aws/credentials ve 

2. ~/.aws/config

isimli iki dosya oluşur

~/.aws/credentials Dosyası - Hassas Bilgileri İçerir

Örnek

Şöyledir

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

~/.aws/credentials Dosyası - Profile ayarlarını içerir

[default] profile için komut seçeneğinde "--profile ..." parametresini belirmeye gerek yoktur

Örnek

Şöyledir

[default]
region=us-west-2
output=json

Kullanım

Örnek

Şöyle yaparız

$ aws configure

(Replace your access key when prompted)
AWS Access Key ID [None]: ABCDEFGHIAZBERTUCNGG 


(Replace your secret key when prompted)
AWS Secret Access Key [None]: uMe7fumK1IdDB094q2sGFhM5Bqt3HQRw3IHZzBDTm 

(you can put your own availability zone here )
Default region name [None]: ap-south-1

Default output format [None]: json

Örnek

Şöyle yaparız

# to setup aws configuration for terraform
aws configure
AWS Access Key ID [None]: <PASTE>
AWS Secret Access Key [None]: <PASTE>

# to see, where are your credentials saved.
cat $HOME/.aws/credentials

configure sso seçeneği

Komutun detayları  burada. Single sign on içindir. 

Örnek - Profile İle Kullanım

Şöyle yaparız

> aws configure sso
SSO start URL [None]: https://foo.awsapps.com/start
SSO Region [None]: us-east-1
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this 
request, open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

ABC-DEFG
The only AWS account available to you is: 123456789012
Using the account ID 123456789012
The only role available to you is: PowerUserAccess
Using the role name "PowerUserAccess"
CLI default client Region [None]: us-east-1
CLI default output format [None]: json
CLI profile name [PowerUserAccess-123456789012]: poweruser

To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile poweruser

Şifrenin tarihi geçince şöyle yaparız

$ aws sso login 

Tüm çıktı şöyle

> aws s3 ls --profile poweruser

The SSO session associated with this profile has expired or is otherwise invalid. 
To refresh this SSO session run aws sso login with the corresponding profile.

> aws sso login --profile poweruser
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this
request, open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

ABCD-EFGH
Successfully logged into Start URL: https://fooo.awsapps.com/start

> aws s3 ls --profile poweruser
...

git worktree seçeneği

Giriş

 git worktree seçeneği bizi IntelliJ ile sürekli branch değiştirmekten kurtarıyor. Çünkü bazen IntelliJ  sürekli bir şeyleri indekslediği için çalışmak zor oluyor

Örnek

Şöyle yaparız

# Add a new worktree
git worktree add ../new-dir existing-branch

# Remove a worktree
git worktree remove ../new-di

Apache Kafka Connect Developer Guide

Giriş

İki tane kavram var

1. Connector

2. Task

Açıklaması şöyle

To copy data between Kafka and another system, users instantiate Kafka Connectors for the systems they want to pull data from or push data to. Connectors come in two flavors: SourceConnectors, which import data from another system, and SinkConnectors, which export data to another system. For example, JDBCSourceConnector would import a relational database into Kafka, and HDFSSinkConnector would export the contents of a Kafka topic to HDFS files.

Implementations of the Connector class do not perform data copying themselves: their configuration describes the set of data to be copied, and the Connector is responsible for breaking that job into a set of Tasks that can be distributed to Kafka Connect workers. Tasks also come in two corresponding flavors: SourceTask and SinkTask. Optionally, the implementation of the Connector class can monitor the data changes of external systems and request task reconfiguration.