Grafana Dashboard Variables

Giriş

Açıklaması şöyle

To add Variables to the dashboard for Customization, From the Dashboard, select dashboard settings on the top and head over to Variables on the left

Şeklen şöyle

Bir örnek şöyle

Docker Compose ve Grafana

Örnek

Şöyle yaparız

grafana:
    image: grafana/grafana-oss:8.5.2
    pull_policy: always
    network_mode: host
    container_name: grafana
    restart: unless-stopped
    links:
      - prometheus:prometheus
    volumes:
      - ./data/grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_SERVER_DOMAIN=localhost

Örnek

Şöyle yaparız

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    depends_on:
      - mysql

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - 3000:3000
    depends_on:
      - prometheus
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana:/var/lib/grafana

Örnek

Şöyle yaparız. Burada Grafana metadata'yı saklamak için PostgreSQL veri tabanını kullanıyor. Bağlanmak için http://<ip_of_the_host_machine>:3111 adresine gideriz.

version: '3.8'
services:
  ...
  pg_grafana:
    container_name: pg_grafana
    image: postgres:15
    restart: always
    environment:
      POSTGRES_DB: my_grafana_db
      POSTGRES_USER: my_grafana_user
      POSTGRES_PASSWORD: my_grafana_pwd
    ports:
      - "5499:5432"
    volumes:
      - pg_grafana:/var/lib/postgresql/data
  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    user: "0:0"
    environment:
      GF_DATABASE_TYPE: postgres
      GF_DATABASE_HOST: pg_grafana:5432
      GF_DATABASE_NAME: my_grafana_db
      GF_DATABASE_USER: my_grafana_user
      GF_DATABASE_PASSWORD: my_grafana_pwd
      GF_DATABASE_SSL_MODE: disable
    restart: unless-stopped
    depends_on:
        - pg_grafana
    ports:
      - 3111:3000
    volumes:
      - grafana:/var/lib/grafana
volumes:
  pg_grafana:
    driver: local
  grafana:
    driver: local

grafana.ini Dosyası

Örnek

Şöyle yaparız

services:
  grafana:
    image: grafana/grafana:10.0.3
    ports:
      - 3000:3000
    volumes:
      - ./grafana/tmp:/var/lib/grafana
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini

Şöyle yaparız

[paths]
data = /var/lib/grafana/data
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
[server]
http_port = 3000

Grafana Dashboard Import

Giriş

Hazır Dashboard'lar burada

Şeklen şöyle

Örnek -  Spring Boot HikariCP / JDBC

6083 numaralı dashboard. Açıklaması burada. Bir örnek burada

Örnek - Kubernetes cluster monitoring (via Prometheus)

Açıklaması burada. Bir örnek burada.

Create the Dashboard

• In Grafana we can create various kinds of dashboards as per our need
• We also have pre-created dashboards in Grafana and we can import them using the Dashboard number.
• For this tutorial we will import one of the pre-created dashboard
• Click on Import and add 3119 (ID of dashboard)
• It will import below dashboard
  

Grafana Kubernetes Deployment

Giriş

1. Önce bir Data Source eklenir. Data Source ConfigMap'in volume olarak yüklenmesi ile olur. ConfigMap'teki data ismi "prometheus.yaml" olmalıdır

2. Deployment yapılır. Deployment tercihen persistent volume kullanır. Böylece ayarlar saklanabilir

Mutlaka olması gereken volume'lar şöyle

1. /var/lib/grafana

2. /etc/grafana/provisioning/datasources

Hazır dashboard kullanmak istiyorsak bunlar şöyle olabilir

1. /etc/grafana/provisioning/dashboards

2. /grafana-dashboard-definitions/0/pods

3. Dashboard yaratılır

Helm

Şöyle yaparız

> helm repo add grafana https://grafana.github.io/helm-charts
> helm pull grafana/grafana --untar --untardir helm

Örnek

Şöyle yaparız. Burada storageClassName bir ortam değişkeni

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: STORAGE_CLASS_TYPE

Deployment için şöyle yaparız. Burada 3 tane ConfigMap kullanılıyor. Birincisi Data Source için. Diğer ikisi de hazır Dashboard için. "configMap:name" için kullanılan isim ile kind: ConfigMap içindeki metadata:name aynı olmalı

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: grafana/grafana:GRAFANA_VERSION
        name: grafana
        ports:
        - containerPort: 3000
          name: http
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - mountPath: /var/lib/grafana
          subPath: grafana
          name: grafana-storage
          readOnly: false
        - mountPath: /etc/grafana/provisioning/datasources
          name: grafana-datasources
          readOnly: false
        - mountPath: /etc/grafana/provisioning/dashboards
          name: grafana-dashboards
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/pods
          name: grafana-dashboard-pods
          readOnly: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 472
      serviceAccountName: grafana
      volumes:
      - persistentVolumeClaim:
          claimName: grafana-storage
        name: grafana-storage
      - name: grafana-datasources
        configMap:
          name: grafana-datasources
      - configMap:
          name: grafana-dashboards
        name: grafana-dashboards
      - configMap:
          name: grafana-dashboard-pods
        name: grafana-dashboard-pods

Data Source için ConfigMap şöyle

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
                "access": "proxy",
                "editable": false,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus-k8s.CUSTOM_NAMESPACE.svc:9090",
                "version": 1
            }
        ]
    }

Dashboardlar için ConfigMap burada

Örnek

Bur örnekte dashboard kopyalanıyor. Deployment şöyle. Burada "grafana-storage" isimli volumeMounts bir PVC'ye atıfta bulunmuyor. Aksine aynı yaml'daki "volume" tanımına atıfta bulunuyor

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests: 
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources

Data Source için ConfigMap şöyle

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus-service.monitoring.svc:8080",
                "version": 1
            }
        ]
    }

Grafana Dashboard Genel Görünüm

Dashboard Genel Görünüm

Örnek bir dashboard şöyle

Açıklaması şöyle

1.Zoom out time range

2. Time picker dropdown. Here you can access relative time range options, auto-refresh options and set custom absolute time ranges.

3. Manual refresh button. Will cause all panels to refresh (fetch new data).

4. Dashboard panel. Click the panel title to edit panels.

5. Graph legend. You can change series colors, y-axis and series visibility directly from the legend.

1. Dashboard'a birden fazla satır eklenebilir.

2. Her satırda bir veya daha fazla panel bulunur. 

Paneller için açıklama şöyle

The panel is the basic visualization building block in Grafana. Each panel has a query editor specific to the data source selected in the panel. The query editor allows you to extract the perfect visualization to display on the panel. With the exception of a few special use panels, a panel is a visual representation of one or more queries. The queries display data over time. This can range from temperature fluctuations to current server status to a list of logs or alerts. In order to display data, needs to have at least one data source added to Grafana.

There are a wide variety of styling and formatting options for each panel. Panels can be dragged and dropped and rearranged on the dashboard. They can also be resized.

Drag and drop panels by clicking and holding the panel title, then dragging it to its new location. it can also easily resize panels by clicking the (-) and (+) icons.

Templates and variables

Açıklaması şöyle

A template is any query that contains a variable.

For example : 

wmi_system_threads{instance=~"$server"}

Variable syntax

Açıklaması şöyle

Panel titles and metric queries can refer to variables using two different syntaxes:

- $varname This syntax is easy to read, but it does not allow users to use a variable in the middle of a word. Example: apps.frontend.$server.requests.count

- ${var_name} Use this syntax when the user wants to interpolate a variable in the middle of an expression.

- ${var_name:<format>} This format gives users more control over how Grafana interpolates values.

- [[varname]] Do not use it. Deprecated old syntax, will be removed in a future release.

Before queries are sent to the data source the query is interpolated, meaning the variable is replaced with its current value. During interpolation, the variable value might be escaped in order to conform to the syntax of the query language and where it is used. For example, a variable used in a regex expression in an InfluxDB or Prometheus query will be regex escaped. Read the data source specific documentation topic for details on value escaping during interpolation.

Variable values are always synced to the URL using the syntax var-<varname>=value.

Variable best practices

- Variable drop-down lists are displayed in the order they are listed in the variable list in Dashboard settings.

-  Put the variables that you will change often at the top, so they will be shown first (far left on the dashboard).

Örnek

Şöyle yaparız. Burada $__timeFilter bir değişken. Grafana dashboard ile seçili zaman aralığını temsil ediyor.

SELECT 
  UNIX_TIMESTAMP(date_format(created_date,'%Y-%m-%d %H:%i')) as time_sec,
  count(*) as value,
  variable_name as metric
FROM dashboard.service_response_time
  WHERE $__timeFilter(created_date)
  GROUP BY time_sec,variable_name
  ORDER BY time_sec ASC;

Dashboard Başlığı

Şeklen şöyle

Açıklaması şöyle

1. Side menubar toggle: This toggles the side menu, allowing you to focus on the data presented in the dashboard. The side menu provides access to features unrelated to a Dashboard such as Users, Organizations, and Data Sources.

2. Dashboard dropdown: This dropdown shows you which Dashboard you are currently viewing, and allows you to easily switch to a new Dashboard. From here you can also create a new Dashboard or folder, import existing Dashboards, and manage Dashboard playlists.

3. Add Panel: Adds a new panel to the current Dashboard

4. Star Dashboard: Star (or unstar) the current Dashboard. Starred Dashboards will show up on your own Home Dashboard by default, and are a convenient way to mark Dashboards that you’re interested in.

5. Share Dashboard: Share the current dashboard by creating a link or create a static Snapshot of it. Make sure the Dashboard is saved before sharing.

6. Save dashboard: The current Dashboard will be saved with the current Dashboard name.

7. Settings: Manage Dashboard settings and features such as Templating and Annotation

Time range controls

Şeklen şöyle

"Last X" şeklinde göreceli veya "Absolute Time" şeklinde mutlak zaman girilebilir.

Grafana Data Source Ekleme

Data Source Olarak AWS Cloud Watch

Add data source ile şöyle yaparız

Bağlantı bilgileri için şöyle yaparız

Açıklaması şöyle

The updated CloudWatch data source ships with pre-configured dashboards for five of the most popular AWS services:

1. Amazon Elastic Compute Cloud Amazon EC2,

2. Amazon Elastic Block Store Amazon EBS,

3. AWS Lambda AWS Lambda,

4. Amazon CloudWatch Logs Amazon CloudWatch Logs, and

5. Amazon Relational Database Service Amazon RDS.

Dashboards sekmesi şeklen şöyle

Data Source Olarak PostgreSQL

Bir örnek burada. Eğer bunu yaparsak PostgreSQL veri tabanına SELECT vs gibi SQL çağrıları yapabiliriz

Data Source Olarak Prometheus

Dashboard ekranından "Add Data Source" seçilir. Şeklen şöyle

veya menü kullanılır. Şeklen şöyle

Prometheus seçilir. Şeklen şöyle

Prometheus ayarları girilir.  Şeklen şöyle. 

Örneğin Prometheus sunucusunun http adresi girilir ve "Save and Test" düğmesi tıklanır. 

Artık dash board ekranından data source Prometheus yapılırsa bilgiler görülebilir ve 

Prometheus ile gelen counter, gauge gibi araçlar ana dashboard ekranına eklenebilir

Örnek

Data Source olarak Prometheus ekledikten sonra ayarları gösteren bir başka şekil şöyle

Grafana

Giriş

Açıklaması şöyle

Grafana is an open source solution for running data analytics, pulling up metrics that make sense of the massive amount of data & to monitor our apps with the help of cool customizable dashboards.
Grafana connects with every possible data source, commonly referred to as databases such as Graphite, Prometheus, Influx DB, ElasticSearch, MySQL, PostgreSQL etc

Açıklaması şöyle

Grafana is a tool that can inject various data sources and display them in a comprehensive graphical experience. It supports various data sources like ELK, Prometheus, Graphite. And It also supports sending alerts base on different conditions.

Prometheus

Grafana, Prometheus'un çıktısını dashboard tarzı daha okunaklı hale getirir. İlişkisi şeklen şöyle

Ekranları gösteren bir örnek burada

Data Source Ekleme

Data Source Ekleme yazısına taşıdım

grafana-server komutu

Grafana'yı başlatmak için "./bin/grafana-server" komutu çalıştırılır

Dashboard

Grafana Dashboard Genel Görünüm yazısına bakabilirsiniz

Grafana Dashboard Import yazısına bakabilirsiniz

Dashboard şöyle. Burada cluster için toplam CPU ve Memory kullanımı görülebilir

Container'lar için toplam CPU ve Memory kullanımı şöyle

Örnek - Create

 create -> Add Panel ile panel eklenir. Her panel'in metrics browser alanına birer birer şunlar eklenir

jenkins_plugins_active{}
jenkins_plugins_inactive{}
jenkins_plugins_failed{}
jenkins_plugins_withUpdate{}

Helm İle Kurulum

Şöyle yaparız

kubectl create ns prometheus
kubectl create ns grafana

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts

# deploy/prometheus-server diye deployment yapar
helm install prometheus prometheus-community/prometheus 
  --namespace prometheus 
  --set alertmanager.persistentVolume.storageClass="gp2" 
  --set server.persistentVolume.storageClass="gp2"

helm install grafana grafana/grafana 
  --namespace grafana 
  --set persistence.storageClassName="gp2" 
  --set persistence.enabled=true 
  --set adminPassword='EKS!sAWSome' 
  --values ./grafana.yaml 
  --set service.type=LoadBalancer

#get the DNS of grafana load balancer.
kubectl get svc -n grafana

Docker

Şöyle yaparız

docker pull grafana/grafana

docker run -d -p 3000:3000 grafana/grafana

"localhost:3000" adresine bağlanıtız. Kullanıcı ismi ve şifre "admin:admin"

Docker Compose

Docker Compose yazısına taşıdım

Kubernetes

Grafana Kubernetes Deployment yazısına taşıdım

Loki

Açıklaması şöyle

Grafana Loki is logging aggregation system which allows high available and multi tenancy.

In micro-service paradigm to read the logs from each micro-service is really hard when we do one by one. To identify one issue have to tail logs in multiple modules. But what loki does is support the log aggregation in single dashboard and will allow to query the results form dashboard.

Şeklen şöyle

promptail-config.yml şöyle. Logları loki'ye gönderir.

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: docker
  entry_parser: raw
  pipeline_stages:
   - docker:{}
  static_configs:
   - labels:
      job: dockerlogs
      __path__: /var/lib/docker/containers/*/*log

promptail için docker plugin kurulmalı. Şöyle yaparız

docker plugin install grafana/loki-docker-driver:latest
  --alias loki --grant-all-permissions

loki-config.yml şöyle

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Şöyle yaparız

version: "3"

networks:
  loki:

services:
  app:
    image: quiz-server:latest
    container_name: 'quiz-server'
    ports:
    - '8080:8080'
  loki:
    image: grafana/loki
    volumes:
      - /home/sajith/Documents/personal/blogs/grafana-loki/loki:/etc/loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /home/sajith/Documents/personal/blogs/grafana-loki/promtail/config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:master
    ports:
      - "3000:3000"
    networks:
      - loki

Influx Veri Tabanı

Influx ile ilişkisi şeklen şöyle