Skip to main content

Otel Collector

src

To collect telemetry data from your services, you will need to setup an OpenTelemetry Collector. This collector will receive data from your services and forward it to your observability stack.

We have not yet defined our observability stack, but we will be using Grafana, Loki, Tempo, Prometheus, and Fluentbit.

Lets get the collector up and running ready to receive data and forward to the observability stack.

Setup​

Create a new directory for the collector and create a docker-compose.yml file.

mkdir ~/local-cloud/otel-collector
cd ~/local-cloud/otel-collector
touch docker-compose.yml

Add the following to the docker-compose.yml file.

networks:
  proxy:
    external: true 

services:
  otel-collector:
    hostname: otelcol
    restart: unless-stopped
    container_name: otelcol
    image: otel/opentelemetry-collector-contrib
    volumes:
      - ./config.yml:/etc/otelcol-contrib/config.yaml
    ports:
      - 1888:1888 # pprof extension
      - 8888:8888 # Prometheus metrics exposed by the Collector
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      # - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
      - 55679:55679 # zpages extension
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.otelcol.rule=Host(`otelcol.localhost`) || Host(`otelcol.dev.localhost`)"
      - "traefik.http.routers.otelcol.entrypoints=web,websecure"
      - "traefik.http.routers.otelcol.tls=true"
      - "traefik.http.routers.otelcol.service=otelcol@docker"
      - "traefik.http.services.otelcol.loadBalancer.server.port=4318"
      - "traefik.http.services.otelcol.loadBalancer.server.scheme=http"

Now we need to make a config file for the collector. Create a config.yml file in the same directory.

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: 0.0.0.0:4318
  # otlp/2:
  #   protocols:
  #     grpc:
  #       endpoint: 0.0.0.0:55690

processors:
  attributes:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: container
      - action: insert
        key: loki.resource.labels
        value: host_name

  batch:
    send_batch_max_size: 10000
    timeout: 10s
exporters:
  # otlp:
    # endpoint: otelcol:4318
  # otlp/2:
  #   endpoint: otelcol2:4317
  # Data sources: traces, metrics
  otlphttp:
    endpoint: http://tempo:4318
  # Data sources: metrics
  prometheusremotewrite:
    # endpoint: http://prometheus.example.com:9411/api/prom/push
    # When using the official Prometheus (running via Docker)
    endpoint: 'http://prometheus:9090/api/v1/write'
    # add:
    tls:
      insecure: true
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
extensions:
  health_check:
  pprof:
  zpages:

service:
  telemetry:
    logs:
      level: DEBUG
      initial_fields:
        service: dt-otelcol
    metrics:
      level: detailed
      address: 0.0.0.0:8888
  extensions: 
    - health_check
    - pprof
    - zpages
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [attributes, batch]
      exporters: [loki]

This will forward data to our observability stack. We will need to setup the observability stack before we can see any data but it should still collect data.

Start the Collector​

Now we can start our collector.

docker compose up -d

You should be able to access the collector at the following URLs.