K8S - Not able to see alerts via - alertmanager

Question

I've Prometheus operator which is working as expected https://github.com/coreos/prometheus-operator

Now I want to apply the alert manager from scratch

After reading the docs im came out with those yamls. but the problem is when I entered to the UI Nothing is shown, any idea what I miss here ?

http://localhost:9090/alerts I use port forwarding ...

This is all the config files I've apply to my k8s cluster I just want to do some simple test to see that it working and then extend it to our needs...

alertmanger_main.yml

---
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: main
  labels:
    alertmanager: main
spec:
  replicas: 3
  version: v0.14.0

alertmanger_service.yml

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-main
spec:
  type: LoadBalancer
  ports:
  - name: web
    port: 9093
    protocol: TCP
    targetPort: web
  selector:
    alertmanager: main

testalert.yml

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-example-rules
  labels:
    role: prometheus-rulefiles
    prometheus: prometheus
data:
  example.rules.yaml: |+
    groups:
    - name: ./example.rules
      rules:
      - alert: ExampleAlert
        expr: vector(1)

alertmanager.yml

global:
  resolve_timeout: 5m
route:
  group_by: ['job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'webhook'
receivers:
- name: 'webhook'
  webhook_configs:
  - url: 'http://alertmanagerwh:30500/'

and to create secret I use

kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml

what I need is some basic alerts in K8S and I follow the documatation but didnt find any good step by step tutorial

to check my sys for monitoring namespace

 ~  kubectl get pods -n monitoring                                                                                                                                                13.4m  Sun Feb 17 18:48:16 2019
NAME                                  READY   STATUS    RESTARTS   AGE
kube-state-metrics-593czc6b4-mrtkb   2/2     Running   0          12h
monitoring-grafana-771155cbbb-scqvx   1/1     Running   0          12h
prometheus-operator-79f345dc67-nw5zc   1/1     Running   0          12h
prometheus-prometheus-0               3/3     Running   1          12h
 ~  kubectl get svc -n  monitoring                                                                                                                                                536ms  Sun Feb 17 21:04:51 2019
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
alertmanager-main   NodePort    100.22.170.666   <none>        9093:30904/TCP   4m53s
kube-state-metrics     ClusterIP   100.34.212.596   <none>        8080/TCP         4d7h
monitoring-grafana     ClusterIP   100.67.230.884   <none>        80/TCP           4d7h
prometheus-operated    ClusterIP   None             <none>        9090/TCP         4d7h

I've also now changed the service to LoadBalancer and I try to enter like

 ~  kubectl get svc -n  monitoring                                                                                                                                                507ms  Sun Feb 17 21:23:56 2019
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE
alertmanager-main   LoadBalancer   100.22.170.666   38.482.152.331   9093:30904/TCP   23m

when I hit the browser with

38.482.152.331:9093 
38.482.152.331:30904 

nothing happen...

Answer 1

When you consider using AlertManager, besides the general configuration and applying alert rules, AlertManager requires being integrated with a Prometheus server. The Prometheus instance can then track any incoming series of events, and once it detects any rule which is recognized, it triggers an alert to the nested alertmanager.

In order to enable alerting it might be necessary to append the following config the to Prometheus instance:

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 'alertmanagerIP:9093'

Specifically, for AlertManager implementation in CoreOS, you can follow the steps described in the official Alerting documentation; however, below you can find example for Prometheus pod alerting configuration kept from the mentioned guideline:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example
spec:
  replicas: 2
  alerting:
    alertmanagers:
    - namespace: default
      name: alertmanager-example
      port: web
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      role: prometheus-rulefiles
      prometheus: example