OpenTelemetry Collector - Sidecar

Örnek

sidecar.yaml şöyledir

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: sidecar
spec:
  mode: sidecar
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
    exporters:
      logging:
      otlp:
        endpoint: "<path_to_central_collector>:4317"
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging, otlp]

instrumentation.yaml  şöyledir

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
spec:
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: always_on
  java:

deployment.yaml şöyledir

apiVersion: apps/v1
kind: Deployment
metadata:
  name: petclinic
  labels:
    app: petclinic
spec:
  replicas: 1
  selector:
    matchLabels:
      app: petclinic
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: 'true'
        sidecar.opentelemetry.io/inject: 'sidecar'
      labels:
        app: petclinic
    spec:
      containers:
      - name: petclinic
        image: <path_to_petclinic_image>
        ports:
        - containerPort: 8080

Açıklaması şöyle

To enable the instrumentation, we need to update the deployment file and add annotations to it. This way we tell the OpenTelemetry Operator to inject the sidecar and the java-instrumentation to our application.

petclinic-svc.yaml şöyledir

apiVersion: v1
kind: Service
metadata:
  name: petclinic-service
spec:
  selector:
    app: petclinic
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

OpenTelemetry Backend

Giriş

Açıklaması şöyle

Even though OpenTelemetry does not provides their own backend, by using it, we are not tied to any tool or vendor, since it is vendor agnostic. Not only can we use any programming language we want, but we can also pick and choose the storage backend and also easily switch to another backend/vendor, by just configure another exporter.

Honeycomb, Lightstep, New Relic, Tempo (Grafana Cloud) gibi bir sürü şey olabilir

Jaeger ve Zipkin

Açıklaması şöyle

Jaeger and Zipkin predate OpenTelemetry, so each has its trace transport format. They do provide integration with the OpenTelemetry format, though.

Jaeger

Açıklaması şöyle

Jaeger inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and can be used for monitoring microservices based distributed systems.

Örnek 

Jaeger'ı çalıştırmak için şöyle yaparız

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

Örnek - Docker Compose ve Jaeger 

Şöyle yaparız

version: "3"

services:
  jaeger:
    image: jaegertracing/all-in-one:1.37           #1
    environment:
      - COLLECTOR_OTLP_ENABLED=true                #2
    ports:
      - "16686:16686"                              #3

Açıklaması şöyle

1. Use the all-in-one image

2. Very important: enable the collector in OpenTelemetry format

3. Expose the UI port

OpenTelemetry Collector

Deployment

İki yöntem var

1. Sidecar

2. A Central (Gateway) OpenTelemetry Collector

1. Sidecar

Açıklaması şöyle

In this scenario, the OpenTelemetry instrumented application sends the data to a (collector) agent that resides together with the application. This agent will then offload responsibility and handle all the trace data from the instrumented application.

The collector can be deployed as an agent via a sidecar which can be configured to send data directly to the storage backend.

Şeklen şöyle

2. A Central (Gateway) OpenTelemetry Collector

Açıklaması şöyle

You can also decide to send the data to another OpenTelemetry collector and from the (central) collector send the data further to the storage backend. In this configuration, we have a central OpenTelemetry collector that is deployed using the deployment mode, which comes with many advantages like auto scaling.

Collector Bileşenleri

Şeklen şöyle

OpenTelemetry Collector üzerinde 3 tane bileşen var. Bunlar

1. Receivers

2. Processors

3. Exporters

OpenTelemetry Protocol

Açıklaması şöyle

OpenTelemetry Protocol (OTLP) specification describes the encoding, transport, and delivery mechanism of telemetry data between telemetry sources, intermediate nodes such as collectors and telemetry backends. 

Each language SDK provides an OTLP exporter you can configure to export data over OTLP. The OpenTelemetry SDK then transforms events into OTLP data.

1. Receivers

Açıklaması şöyle

A receiver, which can be push or pull based, is how data gets into the collector. The OpenTelemetry collector can receive telemetry data in multiple formats.

Örnek - gRPC

Şöyle yaparız

otlp:
  protocols:
    grpc:
      endpoint: "0.0.0.0:4317"

Örnek - gRPC + Http

Şöyle yaparız

otlp:
  protocols:
    grpc:
    http:

2. Processors

Açıklaması şöyle

Processors are run on data between being received and being exported. Processors are optional though some are recommended

batch Processor

Açıklaması şöyle

The batch processor accepts spans, metrics, or logs and places them into batches. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. This processor supports both size and time based batching.

Örnek

Açıklaması şöyle

Configuring a processor does not enable it. Processors are enabled via pipelines within the service section.

Şöyle yaparız

service:
  traces:
    receivers: [opencensus, jaeger]
    processors: [batch]
    exporters: [opencensus, zipkin]
  ...
processors:
  batch:

3. Exporters

Açıklaması şöyle

In order to visualise and analyse the telemetry you will need to use an exporter. An exporter is a component of OpenTelemetry and is how data gets sent to different systems/back-ends.

Açıklaması şöyle. Yani OpenTelemetry protocol (OTLP) formatındaki veriyi başka bir formata çevirir.

Generally, an exporter translates the internal format into another defined format, so you can send different types of data to different backends. For example, you can send metrics (i.e Prometheus) to one backend and traces to another.

Açıklaması şöyle

- The Jaeger exporter is used to send data to Jaeger.

The Logging exporter is very useful when troubleshooting as it exports data to the console

Console Exporter

Açıklaması şöyle

A common exporter to start with and that is very useful for development and debugging tasks is the console exporter.

Örnek

Açıklaması şöyle

In the exporters section, you can add more destinations. For example, if you would also like to send trace data to Grafana Tempo, just add these lines to the central_collector.yaml file.

Şöyle yaparız

pipelines:
  traces:
    receivers: [otlp]
    processors: []
    exporters: [logging, otlp]

exporters:
  logging:
  otlp:
   endpoint: "<tempo_endpoint>"
   headers:
     authorization: Basic <api_token>

4. Extensions

Açıklaması şöyle

Extensions are available primarily for tasks that do not involve processing telemetry data. Examples of extensions include health monitoring, service discovery, and data forwarding. Extensions are optional.

Örnek

Şöyle yaparız

extensions:
  health_check:
  pprof:
  zpages:
  memory_ballast:
    size_mib: 512

OpenTelemetry Java Agent

Giriş

Logları direkt bir backend sunucuya gönderir

Java Agen https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases adresinde

Örnek

Uygulamayı çalıştırmak için şöyle yaparız

java -javaagent:opentelemetry-javaagent.jar -jar catalog.jar 

Örnek

Şöyle yaparız

java -javaagent:opentelemetry-javaagent.jar -jar target/*.jar

Örnek

Şöyle yaparız

# 1. Add them to the startup commands

java -javaagent:path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=your-service-name -jar myApplication.jar

# 2. Use JAVA_TOOL_OPTIONS and other environment variables

export JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar"
export OTEL_SERVICE_NAME="your-service-name"
java -jar myApplication.jar

docker-composer.override.otel.yml

Örnek

Şöyle yaparız

curl --create-dirs -O -L --output-dir ./otel \
https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

docker-composer.override.otel.yml şöyledir

#docker-compose.override.otel.yml
version: '3'

services:
  [your-service]:
  volumes:
      - "./otel/opentelemetry-javaagent.jar:/otel/opentelemetry-javaagent.jar"
  environment:
   - JAVA_TOOL_OPTIONS=-javaagent:/otel/opentelemetry-javaagent.jar
   - OTEL_SERVICE_NAME=[your-service]
   - DEPLOYMENT_ENV=DOCKER_LOCAL
  extra_hosts:
        - "host.docker.internal:host-gateway"

Çalıştırmak için şöyle yaparız

docker compose -f docker-compose.yml -f docker-compose.override.otel.yml up -d

SpringBoot

Örnek

Açıklaması şöyle

OpenTelemtry java agent uses the Mapped Diagnostic Context(MDC) to propagate the TraceID and SpanID within the service. You can print the TraceID and SpanID in the log line by extracting the values from MDC in the console pattern. Just add this to yourapplication.yml file:

Şöyle yaparız

logging:
  level:
    root: INFO
  pattern:
    console: '[%d{yyyy-MM-dd HH:mm:ss.SSS}] [%mdc{trace_id}/%mdc{span_id}] [%thread] %-5level %C:%M:%L - %msg%n'

O zaman çıktı şöyle

[2023–04–14 06:53:02.166] [0eef6546864ff6a0129a4b6ce06f7cbf/c3e97a8c1174afd2] [http-nio-8080-exec-7] INFO ...

Exporter Selection

3 tane değişken tanımlanabilir

1. OTEL_TRACES_EXPORTER     Trace exporter to be used. Default “otlp”

2. OTEL_METRICS_EXPORTER    Metrics exporter to be used. Default “otlp”

3. OTEL_LOGS_EXPORTER     Logs exporter to be used. Default “otlp”

OTEL_TRACES_EXPORTER

Trace'lerin hangi formatta gönderileceğini belirtir. otlp, jaeger, zipkin, none gibi değerler alabilir. Varsayılan değer otlp

Console Backend

Örnek

Şöyle yaparız. opentelemetry-javaagent-all.jar opentelemetry agent için jar.Servisimizi ismi my-service

OTEL_SERVICE_NAME=my-service \
OTEL_TRACES_EXPORTER=logging \
java -javaagent:./opentelemetry-javaagent.jar \
     -jar target/*.jar

Çıktısı şöyle

//Console output

INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - '/owners/{ownerId}' :

99af87eeaa19b83d014463b046884e56 632e2d2a5931bd17 SERVER [tracer: io.opentelemetry.tomcat-7.0:1.13.1-

alpha] AttributesMap{data={net.transport=ip_tcp, http.target=/owners/11, thread.id=120, http.flavor=1.1,

http.status_code=200, net.peer.ip=0:0:0:0:0:0:0:1, thread.name=http-nio-8080-exec-10,

http.host=localhost:8080, http.route=/owners/{ownerId}, http.user_agent=Mozilla/5.0 (Macintosh; Intel

Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36,

http.method=GET, net.peer.port=50778, http.scheme=http}, capacity=128, totalAddedValues=13}

Jaeger Backend

Jaeger'a özel ayarların açıklaması şöyle

OTEL_EXPORTER_JAEGER_ENDPOINT : Full URL of the Jaeger HTTP endpoint

OTEL_EXPORTER_JAEGER_TIMEOUT : Maximum time (in milliseconds) the Jaeger exporter will wait for each batch export

OTEL_EXPORTER_JAEGER_USER : Username to be used for HTTP basic authentication

OTEL_EXPORTER_JAEGER_PASSWORD : Password to be used for HTTP basic authentication

Örnek

Şöyle yaparız

OTEL_SERVICE_NAME=my-service 
OTEL_TRACES_EXPORTER=jaeger 
OTEL_EXPORTER_JAEGER_ENDPOINT=http://localhost:14250 \
  java -javaagent:./opentelemetry-javaagent.jar -jar target/*.jar

Aspecto Backend

Örnek

Şöyle yaparız. Bu sefer OTEL_TRACES_EXPORTER tanımlamaya gerek yok. Varsayılan değer otlp ve Aspecto da bu formatı destekliyor. Sadece OTEL_EXPORTER_OTLP_TRACES_ENDPOINT yeterli

OTEL_SERVICE_NAME=my-service 
OTEL_EXPORTER_OTLP_HEADERS=Authorization={ASPECTO_AUTH}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otelcol.aspecto.io:4317 \
  java -javaagent:./opentelemetry-javaagent.jar \
       -jar target/*.jar

Lightstep Backend

Lightstep normal opentelemetry-javaagent.jar'ı sarmalayan bir başka jar lightstep-opentelemetry-javaagent.jar isimli bir başka jar sağlıyor

Örnek

Şöyle yaparız. Bu sefer OTEL_TRACES_EXPORTER tanımlamaya gerek yok. Varsayılan değer otlp ve Lightstep de bu formatı destekliyor. Sadece OTEL_EXPORTER_OTLP_TRACES_ENDPOINT yeterli

export LS_ACCESS_TOKEN=your-token
export OTEL_SERVICE_NAME=springboot-demo
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.lightstep.com:443

java -javaagent:lightstep-opentelemetry-javaagent.jar -jar target/*.jar

Örnek

Şöyle yaparız. Bu sefer OTEL_TRACES_EXPORTER tanımlamaya gerek yok. Varsayılan değer otlp ve Lightstep de bu formatı destekliyor. Ayrıca OTEL_EXPORTER_OTLP_ENDPOINT değeri de belirtilmiyor çünkü varsayılan değer tanımlı O da http://localhost:4318.  Yani yerel bilgisayarda Lightstep çalışıyor.

export OTEL_SERVICE_NAME=springboot-demo_jaeger
export LIGHTSTEP_ACCESS_TOKEN="<your_ls_access_token"

java -javaagent:opentelemetry-javaagent.jar -jar target/*.jar

Örnek

Şöyle yaparız

export LS_ACCESS_TOKEN=your-token
export OTEL_SERVICE_NAME=springboot-demo
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.lightstep.com:443

OpenTelemetry Nedir?

OpenTelemetry Nedir?

Açıklaması şöyle. Yani Open Tracing ve OpenCensus projelerinin birleşmesinden ortaya çıkmıştır.

The OpenTelemetry website states that:

"OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software's performance and behavior."

OpenTelemetry was created by merging the popular OpenTracing and OpenCensus projects. It is a standard that integrates with many open source and commercial products written in many programming languages. Implementations of OpenTelemetry are in varying stages of maturity.

At its core, OpenTelemetry contains the Collector, a vendor-agnostic way to receive, process, and export telemetry data.

OpenTelemetry Nasıl Çalışır

- Uygulama OpenTelemetry SDK ile instrument edilir.

- Artık loglar bir OpenTelemetry Collector'a gönderilir. 

- Collector üzerinde gerekli filtrelemeler yapılır ve 

- Collector logları bir backend'e gönderir.

Şeklen şöyle

instrumentation-annotations

Maven

Şu satırı dahil ederiz

<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-instrumentation-annotations</artifactId>
  <version>1.17.0-alpha</version>
</dependency>

Örnek

Şöyle yaparız

import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan

@WithSpan("ProductHandler.fetch")
suspend fun fetch(@SpanAttribute("id") id: Long): Result<Product> {
  ...
}

OpenTelemetry Backend Nedir?

OpenTelemetry Backend yazısına taşıdım

OpenTelemetry Operator

Şöyle yaparız

kubectl apply \
-f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml