Using alert annotations in an alertmanager receiver

Question

I've got an alert configured like this:

ALERT InstanceDown
IF up == 0
FOR 30s
ANNOTATIONS {
    summary = "Server {{ $labels.Server }} is down.",
    description = "{{ $labels.Server }} ({{ $labels.job }}) is down for more than 30 seconds."
}

The slack receiver looks like this:

receivers:
- name: general_receiver
  slack_configs:
  - api_url: https://hooks.slack.com/services/token
    channel: "#telemetry"
    title: "My summary"
    text: "My description"

Is it possible to use the annotations in my receiver? This github comment indicates it is but I haven't been able to get anything to work from it.

Answer 1

You need to define your own template (I just hat to walk that path). For a brief example see: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

All alerts here have at least a summary and a runbook annotation. runbook contains an Wiki URL. I've defined the following template (as described in the blog article above) to include them into the slack message body:

{{ define "__slack_text" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.summary }}                                                                                                                                    
{{ if gt (len .Labels) (len .GroupLabels) }}({{ with .Labels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}                                             
{{ end }}<{{ (index .Alerts 0).GeneratorURL }}|Source> | {{ if .CommonAnnotations.runbook }}<{{ .CommonAnnotations.runbook }}|:notebook_with_decorative_cover: Runbook>{{ else }}<https://wiki.some.where/Runbooks|:exclamation:*NO RUNBOOK*:exclamation:>{{ end }}                                                                                
{{ end }}   
{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}  

The templates overwrites the slack.default.text so there is no need to reference it in the receiver configuration.

Defaults can be found in source as well as the "documentation":

• https://github.com/prometheus/alertmanager/blob/v0.4.2/template/default.tmpl
• https://github.com/prometheus/alertmanager/blob/v0.4.2/template/template.go

For basics and details about the golang templating language:

• https://gohugo.io/templates/go-templates/
• https://golang.org/pkg/text/template/

For completeness' sake, the eventual template:

{{ define "__slack_text" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.text }}{{ end }}                                                                                
{{ end }}

{{ define "__slack_title" }}                                                                                                                                                
{{ range .Alerts }}{{ .Annotations.title }}{{ end }}                                                                                
{{ end }}

{{ define "slack.default.text" }}{{ template "__slack_text" . }}{{ end }}
{{ define "slack.default.title" }}{{ template "__slack_title" . }}{{ end }}

Answer 2

I know this response is years later, but my approach was to leverage the .CommonAnnotations and labels to display the information, and the alertname to query for a runbook -

alertmanager.yml

  slack_configs:
  - api_url: 
    channel: '#XXXXXXXXXXXXXXXX'
    color: '{{ template "SLACK_MSG_COLOR" . }}'
    send_resolved: true
    title: '{{ template "SLACK_MSG_TITLE" . }}'
    text: '{{ template "SLACK_MSG_TEXT" . }}'


  pagerduty_configs:
  - routing_key: '{{ template "Global_PD_Service_Key" . }}'
    description: '{{ template "PAGERDUTY_DESCRIPTION" . }}'
    severity: '{{ if .CommonLabels.severity }}{{ .CommonLabels.severity | toLower }}{{ else }}critical{{ end }}'
    links: 
    - text: 'Prometheus'
      href: '{{ (index .Alerts 0).GeneratorURL }}'
    - text: 'Search Runbooks'
      href: '{{ template "RUNBOOK_SEARCH" . }}'

and the .tmpl

################  
# Runbook  
################ 
# Runbook Search
{{ define "RUNBOOK_SEARCH" }}https://dsmith73.github.io/101-docs/search/?q={{ .CommonLabels.alertname }}{{ end }}



################  
# Slack  
################  
# Slack Color
{{ define "SLACK_MSG_COLOR" }}{{ if eq .Status "firing" }}{{ if eq .CommonLabels.severity "critical" }}danger{{ else if eq .CommonLabels.severity "error" }}danger{{ else if eq .CommonLabels.severity "warning" }}warning{{ else }}#439FE0{{ end }}{{ else}}good{{ end }}{{ end }}


# Slack Text  
{{define "SLACK_MSG_TEXT" }}
      <!here> - {{ .CommonAnnotations.description }}
      `View:` :chart_with_upwards_trend:*<{{ (index .Alerts 0).GeneratorURL }}|Prometheus>* or :notebook:*<{{ template "RUNBOOK_SEARCH" . }}|Runbook>*

      *Details:*
      {{ range .CommonLabels.SortedPairs }}• *{{ .Name }}:* `{{ .Value }}`
    {{ end }}
{{ end }}

#Slack Summary
{{ define "SLACK_TITLE_SUMMARY" -}}
    {{- if .CommonAnnotations.summary -}}
        {{- .CommonAnnotations.summary -}}
    {{- else -}}
        {{- with index .Alerts 0 -}}
            {{- .Annotations.summary -}}
        {{- end -}}
    {{- end -}}
{{- end -}}


# Slack Title  
{{ define "SLACK_MSG_TITLE" }}
    {{ if eq .Status "resolved" }}
        {{- .Status | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
    {{ else if eq .Status "firing" }}
        {{ .CommonLabels.severity | toUpper }} : {{ template "SLACK_TITLE_SUMMARY" . }}
    {{ end }}
{{ end }}

To me, this was a nice way to tie alerts to runbooks, rather than defining them in rules, or other locations...