Elastic APM HTTP error (503): queue is full

Question

Kibana response is

{
"statusCode": 500,
"error": "Internal Server Error",
"message": "An internal server error occurred"
}

APM server returns 503 - Internal Server Error, Having hard time identifying the root cause. Is it ES queue is full or ran out of memory or cluster is not being setup correctly?

According to ES documentation: https://www.elastic.co/guide/en/apm/server/master/common-problems.html#queue-full

A full queue generally means that the agents collect more data than APM server is able to process. This might happen when APM Server is not configured properly for the size of your Elasticsearch cluster, or because your Elasticsearch cluster is underpowered or not configured properly for the given workload.

The queue can also fill up if Elasticsearch runs out of disk space.

Documentation doesn't help in identifying what could be the root cause. How do we identify the root cause?

Restarting Kibana and Elasticsearch helps but that does not help to identify the root cause

Answer 1

You can try to increase:

• Internal queue size (queue.mem.events)
• Elasticsearch bulk size (output.elasticsearch.bulk_max_size).

Please take a look on documentation: Tune APM Server