Monitoring Docker containers using Prometheus + cAdvisor + Grafana

With the increasing adoption of containers and microservices in the enterprises, monitoring utilities have to handle more services and server instances than ever before. Support for multi-dimensional data collection, querying and perfect dashboard visualization tool is a great strength and makes the right fit to be part of your operational toolset. In this post, we take look at  Monitoring Docker containers using Prometheus + cAdvisor + Grafana.

This quickstart assumes basic understanding of Docker concepts, please refer to earlier posts for understanding on Docker & how to install and containerize applications.

Architecture

We are going to use the following tools to collect, aggregate & visualize metrics.

1. Container metrics using cAdvisor
2. Host metrics using Prometheus Node Exporter
3. Alerting using Prometheus Alert Manager
4. Visualization using Grafana pre-built dashboards

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints, we are going to configure it to scrape metrics for containers from cAdvisor and node metrics from Prometheus Node exporter.

#1. Prepare Docker Compose Configuration

Create New Project directory and prepare Docker compose file like the one below

version: '3'
services:
prometheus:
    image: 'prom/prometheus:v2.1.0'
    container_name: prometheus
    ports:
    - '9090:9090'
    command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--web.console.libraries=/usr/share/prometheus/console_libraries'
    - '--web.console.templates=/usr/share/prometheus/consoles'
    volumes:
    - './prometheus.yml:/etc/prometheus/prometheus.yml:ro'
    - './alert.rules:/etc/prometheus/alert.rules'
    depends_on:
    - cadvisor
    networks:
    - samplenet
cadvisor:
    image: google/cadvisor
    container_name: cadvisor
    ports:
    - '8081:8081'
    volumes:
    - '/:/rootfs:ro'
    - '/var/run:/var/run:rw'
    - '/sys:/sys:ro'
    - '/var/lib/docker/:/var/lib/docker:ro'
    networks:
    - samplenet
alertmanager:
    image: prom/alertmanager
    ports:
    - '9093:9093'
    volumes:
    - './alertmanager/:/etc/alertmanager/'
    restart: always
    command:
    - '--config.file=/etc/alertmanager/config.yml'
    - '--storage.path=/alertmanager'
    networks:
    - samplenet
node-exporter:
    image: prom/node-exporter
    volumes:
     - '/proc:/host/proc:ro'
     - '/sys:/host/sys:ro'
     - '/:/rootfs:ro'
    command:
     - '--path.procfs=/host/proc'
     - '--path.sysfs=/host/sys'
     - '--collector.filesystem.ignored-mount-points'
     - ^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)
    ports:
     - '9100:9100'
    networks:
     - samplenet
    restart: always
grafana:
    image: grafana/grafana
    depends_on:
    - prometheus
    ports:
    - '3000:3000'
    volumes:
    - 'grafana_data:/var/lib/grafana'
    - './grafana/provisioning/:/etc/grafana/provisioning/'
    env_file:
    - ./grafana/config.monitoring
    networks:
    - samplenet
    restart: always
volumes:
grafana_data: {}
prometheus_data: {}

#2.Prometheus Node Exporter Configuration

On the compose file, mount the host /proc and /sys directory so that the container has access to the necessary information to report on.

#3.cAdvisor Configuration

On the compose file, mount the var/run, /sys directory /var/lib/docker so that the cAdvisor can collect container metrics and report to Prometheus.

#4.Prometheus Configuration

Create new Prometheus configuration in a file called prometheus.yml . Prometheus server requires a configuration file that defines the endpoints to scrape along with how frequently the metrics should be accessed and to define the servers and ports that Prometheus should scrape data from. In the below example, we have defined 3 targets (1st one for Prometheus itself, 2nd One for collecting container metrics using cAdvisor and last one is for node metrics using Node exporter) running on different ports.

scrape_configs:
- job_name: prometheus
scrape_interval: 5s
static_configs:
- targets:
- prometheus:9090

- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets:
- cadvisor:8081

- job_name: node-exporter
scrape_interval: 5s
static_configs:
- targets:
- node-exporter:9100

alerting:
alertmanagers:
- static_configs:
- targets: 
- alertmanager:9093

rule_files:
- 'alert.rules'

For a complete specification of configuration options, see the configuration documentation.

#5.Alert Manager Configuration

Alert Manager takes care of de-duplicating, grouping, and routing the alerts to the correct receiver integration such as email, slack channels, etc.,

We are going to make use of default configuration like the example below, create a new folder alertmanager and paste the configuration file (config.yml)

route:
   receiver: 'slack'

receivers:
   - name: 'slack'
      slack_configs:
      - send_resolved: true
      username: '<username>'
      channel: '#<channel-name>'
      api_url: '<incomming-webhook-url>'

#6.Grafana Configuration

Grafana would be the dashboard visualization tool of choice for Prometheus users and support for Grafana ships with the tool. For monitoring Docker containers, we are going to import pre-built dashboards from Grafana.com.

Create new folder Grafana and copy the contents from the source to automate the provisioning of data sources & dashboards. For the full Grafana installation instructions, see the official Grafana documentation.

Finally, our folder would look like the one below.

Congrats! we have now configured all tools to monitor our containers, let’s start the compose and check.

#7.View Metrics

Start the Docker compose using docker-compose up -d

Once all the containers are up, Prometheus will now scrape and store the data based on the configuration. Prometheus is configured on port 9090, Go to the dashboard http://localhost:9090 and verify that Prometheus now has information about the time series information on the containers, node.

Use the dropdown next to the “Execute” button to see a list of metrics this server is collecting. In the list, you’ll see a number of metrics prefixed with node_, that have been collected by the Node Exporter. For example, you can see the node’s CPU usage via the node_cpu metric.

With this expression browser, you can enter any expression and see its result either in a table or graph over time.

Grafana will be listening on http://localhost:3000. The default login is "admin" / "admin", once you log in use the “Filter” option to browse for dashboards. From the list, choose “Docker Prometheus Monitoring” to view it.

If you want to modify Dashboard or data source, you should manually edit the downloaded JSON files and correct the datasource: entries to reflect the Grafana data source name which you chose for your Prometheus server. Use the “Dashboards” → “Home” → “Import” option to import the edited dashboard file into your Grafana install.

All source files used for this article can be found here.

In this post, we have got introduced on how to set up targets in Prometheus, configure Prometheus, cAdvisor, Node Exporter and Alert Manager to monitor both containers and host resources.

Like this post? Don’t forget to share it!

Additional Resources

• Prometheus Documentation
• Prometheus Configuration options
• Prometheus Guides
• Prometheus Tutorial
• Google cAdvisor for monitoring your containers + Docker tutorial
• TOP 6 GUI tools for managing Docker environments
• Docker tutorial – Build Docker image for your Java application