How do I delete a time series from Prometheus v2, specifically a series of alerts

Question

We are getting to grips with alerting so from time to time need to clear out old alerts which we did by calling the HTTP API, to remove the pseudo time series where the alerts were stored, e.g.:

DELETE https://prometheus/api/v1/series?match[]={__name__="ALERTS"}

We have recently upgraded our Prometheus server from 1.8 to 2.2.1.

Calling this endpoint now gives

{
    "status": "error",
    "errorType": "internal",
    "error": "not implemented"
}

I have done some research and found a solution in various locations, which I will summarise in an answer below in case it's of use to my fellow StackOverflowers

Answer 1

Firstly the admin API is not enabled by default in Prometheus 2. This must be made active by starting the server with the option

--web.enable-admin-api

There is a new endpoint in v2 at

https://prometheus/api/v2/admin/tsdb/delete_series

This takes a POST specifying the search criteria, e.g. for a time series with the name ALERTS where the alert name is MyTestAlert, post the following application/json to the delete_series endpoint, from the tool of choice (Tested with Postman 6 on Mac)

{
    "matchers": [{
        "type": "EQ",
        "name": "__name__",
        "value": "ALERTS"
    },
    {
        "type": "EQ",
        "name": "alertname",
        "value": "MyTestAlert"
    }]
} 

For completeness and to free the disk space where the alerts were persisted, POST an empty payload to

https://prometheus/api/v2/admin/tsdb/clean_tombstones

Answer aggregated from:

• https://prometheus.io/docs/prometheus/latest/querying/api/
• https://github.com/prometheus/prometheus/issues/3584
• https://groups.google.com/forum/#!msg/prometheus-users/ToMKsb9fYp8/az6afuX3CgAJ