loki: "error at least one label pair is required per stream"

Question

I am running loki, pomtrail, grafana locally in docker per these instructions

When I try to run the following test to send data directly to loki:

curl -i -H "Content-type: application/json" -X POST --data '{ "streams": [ { "labels": { "job": "randomjob" }, "entries": [{ "ts": "2021-10-12T16:13:06.801064Z", "line": "TEST!" }] } ] }' http://localhost:3100/loki/api/v1/push

I get the error:

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 02 Nov 2021 12:09:26 GMT
Content-Length: 53

error at least one label pair is required per stream

Why isn't if finding the label? Thanks.

Answer 1

Format of the label pairs should be (for an old endpoint /api/prom/push) and it works:

"labels": "{job=\"randomjob\"}"

E.g.:

curl -H "Content-Type: application/json" -XPOST -s "172.30.11.21:3100/api/prom/push" --data-raw   '{"streams": [
  { 
      "labels": "{job=\"randomjob\"}", 
      "entries": [
          { 
              "ts": "2021-11-13T19:55:51.801064-00:00", 
              "line": "TEST!" 
          }
      ] 
  }
]

}'

For a new endpoint /loki/api/v1/push it seems need to change labels field to stream:

curl -H "Content-Type: application/json" -XPOST -s \
  "172.30.11.21:3100/loki/api/v1/push" --data-raw \
  '{"streams": [{ "stream": {"job": "randomjob"}, "values": [["1636841990000000000",  "TEST!" ]] }] }]}'

Answer 2

This error happens if no labels are generated. This can happen e.g. during dropping a message in relabel_configs resulting in {} as a message for loki.

sudo /usr/bin/promtail -config.file /etc/promtail/config.yml -config.expand-env=true -inspect -dry-run -log.level=debug

Can help to see what is send to Loki.