Filebeat

Filebeat

Önce neden Filebeat gerektiğine bakalım. Açıklaması şöyle. Yani Logstash'in performans sıkıntısı yüzünden ortaya çıkmış.

The original task of Logstash is monitoring logs and transforming them into a meaningful set of fields and eventually streaming the output to a defined destination. However, it has an issue with performance.

So, Elastic has launched Filebeat that use for monitoring logs and streaming the output to a defined destination.

And Logstash acts as an aggregator that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.”

Açıklaması şöyle.

Filebeat: Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.

Şeklen şöyle. Tüm container'ları dinler ve logları Logstash'e veya direkt Elastic'e gönderir

1. Docker Compose

Filebeat için bir tane filebeat.yaml dosyası belirtmek gerekir. Şöyle yaparız

filebeat:
    image: elastic/filebeat:8.0.1
    user: root
    volumes:
      - ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - type: bind
        source: /var/lib/docker/containers
        target: /var/lib/docker/containers // Listen to running container files
        read_only: true
      - type: bind
        source: /var/run/docker.sock 
        target: /var/run/docker.sock // It's the main entry point for Docker API.
        read_only: true
depends_on:
      - logstash

filebeat.yaml Dosyası 

Örnek  - logstash

Şöyle yaparız. Docker loglarını Logstash'e gönderir. 

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

output.logstash: 
  hosts: ["logstash:5000"]

logging.level: error

Örnek  - logstash

Şöyle yaparız. Docker loglarını Logstash'e gönderir. 

filebeat.inputs:
- type: log
  paths:
    - "myapp.log"

output.logstash:
  hosts: ["192.168.1.1:4561"]

Örnek - ElasticSearch

Şöyle yaparız. Docker loglarını ElasticSearch' e gönderir. 

output.elasticseach: 
  hosts: ["elasticsearch:5000"]

Örnek - ElasticSearch

Şöyle yaparız. Docker loglarını ElasticSearch' e gönderir. 

filebeat.inputs:
- type: log
  paths:
    - "myapp.log"

output.elasticsearch:
  hosts: ["http://192.168.1.1:9200"] 

Örnek

Şöyle yaparız. Bizim json dosyalarımızı ElasticSearch' e gönderir.

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /metrics/query-output.log
  json.add_error_key: true
  fields:
    type: "db.metrics"

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

2. Kubernetes

Örnek

Elimizde bir ConfigMap olsun. /var/log/*.log dosyasından okur ve Logstash'e gönderir

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    component: filebeat
data:
  conf.yaml: |
    filebeat.inputs:
    - type: log
      paths:
        - '/var/log/*.log'
    output:
      logstash:
        hosts: [ "logstash:5044" ]

Şöyle yaparız

apiVersion: apps/v1
kind: Deployment
...
  spec:
    containers:
      ...
      - name: filebeat
        image: elastic/filebeat:7.16.3
        args:
          - -c
          - /etc/filebeat/conf.yaml
          - -e
        volumeMounts:
          - name: filebeat-config
            mountPath: /etc/filebeat
          - name: log
            mountPath: /var/log
    volumes:
      - name: log
        emptyDir: {}
      - name: filebeat-config
        configMap:
          name: filebeat-config