RabbitMQ lager_error_logger_h dropped messages

Question

Help please solve the problem. There are: RabbitMQ - 3.7.2 Erlang - 20.1

• Connections: 527
• Channels: 500
• Exchanges: 49
• Queues: 4437
• Consumers: 131
• Publish rate ~ 200/s
• Ack rate ~ 200/s

Config:

disk_free_limit.absolute = 5GB
log.default.level = warning
log.file.level = warning

In the logs constantly appear such messages:

11:42:16.000 [warning] <0.32.0> lager_error_logger_h dropped 105 messages in the last second that exceeded the limit of 100 messages/sec
11:42:17.000 [warning] <0.32.0> lager_error_logger_h dropped 101 messages in the last second that exceeded the limit of 100 messages/sec
11:42:18.000 [warning] <0.32.0> lager_error_logger_h dropped 177 messages in the last second that exceeded the limit of 100 messages/sec

How to get rid of them correctly? How to remove this messages from logs?

Answer 1

The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

---

The message means that RabbitMQ is generating a very large number of error messages and that they are being dropped to avoid filling the log rapidly. If "dropped X messages in the last second" is the only message you are seeing in the logs, you need to determine what the messages are that are being dropped to find the root of the problem. You can do this by temporarily raising that limit by running the following command:

rabbitmqctl eval '[lager:set_loghwm(H, 250) || H <- gen_event:which_handlers(lager_event)].'

You should then see a much larger number of messages that will reveal the underlying issue. To revert back to the previous setting, run this command:

rabbitmqctl eval '[lager:set_loghwm(H, 50) || H <- gen_event:which_handlers(lager_event)].'