Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

nginx logs are out of order, probably due to buffered logging

Tags:

nginx

Records in my nginx log file are out of order. (Edit: by "out of order" I mean chronologically. e.g. Log lines for 2017-02-21 09:13:26 will often be before lines for 2017-02-21 09:13:45) Perhaps a certain amount of out of order records are to be expected because they are logged after a request is completed, not when received. But this is a way higher number of requests that are being logged out of order, including known short (fast) requests for small static files.

Is this a known side effect of using buffered logging or can this be improved?

For getting a more complete picture, here are some other config params:

In nginx.conf:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' 

In the config file for the virtual host:

server {
    #The backlog parameter matches sysctl net.core.somaxconn setting. Default value is 511 on Ubuntu.
    listen 80 backlog=30000;
    server_name  www.example.com;
    access_log  /var/log/nginx/access.log main buffer=128k;
    error_log   /var/log/nginx/error.log;
    root   /var/www/html/website;
    ...
}
like image 751
tinkerr Avatar asked Jan 27 '17 19:01

tinkerr


1 Answers

Yes, it's buffering issue. If you are using few workers - each worker has own buffer.

Ways to improve:

  • disable buffering
  • decrease buffer size (1)
  • add flush options, if flush to disk still rare
  • create own log collector with sorting (nginx can syslog protocol, for example)

But usually you don't need to care about order of log records. Log analytic systems will sort it by self.

(1) For linux systems buffer size must not exceed the size of an atomic write to a disk file. In modern linux - it's 64k. Well, I'm not 100% sure about this size because information very discrepant. But if you will find broke lines in log - decrease this size.

like image 73
Dmitry MiksIr Avatar answered Sep 28 '22 06:09

Dmitry MiksIr