How to configure prometheus with alertmanager?

Question

docker-compose.yml: This is the docker-compose to run the prometheus, node-exporter and alert-manager service. All the services are running great. Even the health status in target menu of prometheus shows ok.

version: '2'

services:

    prometheus:
        image: prom/prometheus
        privileged: true
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - ./alertmanger/alert.rules:/alert.rules
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'

    node-exporter:
        image: prom/node-exporter
        ports:
            - '9100:9100'

    alertmanager:
        image: prom/alertmanager
        privileged: true
        volumes:
             - ./alertmanager/alertmanager.yml:/alertmanager.yml
        command:
            - '--config.file=/alertmanager.yml'
        ports:
            - '9093:9093'

prometheus.yml

This is the prometheus config file with targets and alerts target sets. The alertmanager target url is working fine.

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'my-monitor'

# this is where I have simple alert rules
rule_files:
  - ./alertmanager/alert.rules

scrape_configs:
    - job_name: 'prometheus'
      static_configs: 
        - targets: ['localhost:9090']

    - job_name: 'node-exporter'
      static_configs:
        - targets: ['node-exporter:9100']

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['some-ip:9093']

alert.rules: Just a simple alert rules to show alert when service is down

ALERT service_down
  IF up == 0

alertmanager.yml

This is to send the message on slack when alerting occurs.

global:
  slack_api_url: 'https://api.slack.com/apps/A90S3Q753'

    route:
      receiver: 'slack'

    receivers:
      - name: 'slack'
        slack_configs:
          - send_resolved: true
            username: 'tara gurung'
            channel: '#general'
            api_url: 'https://hooks.slack.com/services/T52GRFN3F/B90NMV1U2/QKj1pZu3ZVY0QONyI5sfsdf'

Problems: All the containers are working fine I am not able to figure out the exact problem.What am I really missing. Checking the alerts in prometheus shows.

Alerts No alerting rules defined

Answer 1

Your ./alertmanager/alert.rules file is not included in your docker config, so it is not available in the container. You need to add it to the prometheus service:

prometheus:
    image: prom/prometheus
    privileged: true
    volumes:
        - ./prometheus.yml:/etc/prometheus/prometheus.yml
        - ./alertmanager/alert.rules:/alertmanager/alert.rules
    command:
        - '--config.file=/etc/prometheus/prometheus.yml'

    ports:
        - '9090:9090'

And probably give an absolute path inside prometheus.yml:

rule_files:
- "/alertmanager/alert.rules"

You also need to make sure you alerting rules are valid. Please see the prometheus docs for details and examples. You alert.rules file should look something like this:

groups:
- name: example
  rules:

  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m

Once you have multiple files, it may be better to add the entire directory as a volume rather than individual files.