Skip to main content

Prometheus

src

Prometheus is an open-source monitoring and alerting toolkit that is used for recording real-time metrics in a time-series database. It is designed for reliability, scalability, and ease of use.

We will only need to manage its configuration file and the docker-compose file to get it up and running.

Setup​

Create a new directory for Prometheus and create a docker-compose.yml file.

mkdir ~/local-cloud/prometheus
cd ~/local-cloud/prometheus
touch docker-compose.yml

Add the following to the docker-compose.yml file.

volumes:
  data:

networks:
  proxy:
    external: true

services:
  prometheus:
    container_name: prometheus 
    hostname: prometheus
    restart: unless-stopped
    networks:
      - proxy
    image: prom/prometheus:v2.45.6
    command:
      - --config.file=/etc/prometheus.yml
      - --web.enable-remote-write-receiver
      - --enable-feature=exemplar-storage
    volumes:
      - ./prometheus.yml:/etc/prometheus.yml
      # https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
      - data:/prometheus
    labels:
    - "traefik.enable=true"
    - "traefik.http.routers.prometheus.rule=Host(`prometheus.localhost`) || Host(`prometheus.dev.localhost`)"
    - "traefik.http.routers.prometheus.entrypoints=web,websecure"
    - "traefik.http.routers.prometheus.tls=true"
    - "traefik.http.routers.prometheus.service=prometheus@docker"
    - "traefik.http.services.prometheus.loadBalancer.server.port=9090"

Prometheus Configuration​

Now we need to make a config file for Prometheus. Create a prometheus.yml file in the same directory.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: [ 'localhost:9090' ]
  - job_name: 'tempo'
    static_configs:
      - targets:
        - 'tempo:3200'
        # - 'tempo2:3200'
        # - 'tempo3:3200'

  - job_name: 'loki'
    static_configs:
      - targets:
        - 'loki:3100'

  # - job_name: 'traefik'
  #   tls_config:
  #     insecure_skip_verify: true
  #   static_configs:
  #     - targets:
  #       - 'traefik'

  - job_name: 'grafana'
    static_configs:
      - targets:
        - 'grafana:3000'

  # Example job for node_exporter
  # - job_name: 'node_exporter'
  #   static_configs:
  #     - targets: ['node_exporter:9100']

  # Example job for cadvisor
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

Scrape Configurations​

Job configurations are used to scrape metrics from different services. Each job has a list of targets that are scraped at a given interval.

  • scrape_interval: Interval to scrape metrics
  • evaluation_interval: Interval to evaluate rules
  • static_configs: List of targets to scrape
  • job_name: Name of the job
  • targets: List of targets to scrape

To add a target you must update the prometheus.yml file and add a new job configuration.

  - job_name: 'new_job'
    static_configs:
      - targets:
        - 'new_job:port'

Then Restart the service

Start Prometheus​

docker compose up -d

You should now be able to access Prometheus at http://prometheus.localhost or http://prometheus.dev.localhost

Cadvisor​

If you want to monitor the resource usage of your containers, you can use cAdvisor. Add the following to your docker-compose.yml file.


services:
  ...
  
  cadvisor:
    image: cadvisor/cadvisor:latest    
    container_name: cadvisor
    hostname: cadvisor
    networks:
    - proxy 
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    privileged: true
    restart: unless-stopped

Then Restart the service

docker compose up -d

Prometheus will now scrape metrics from cAdvisor.

Conclusion​

View Prometheus UI at