Logstash close file descriptors?

Question

BACKGROUND:

We have rsyslog creating log files directories like: /var/log/rsyslog/SERVER-NAME/LOG-DATE/LOG-FILE-NAME So multiple servers are spilling out their logs of different dates to a central location.

Now to read these logs and store them in elasticsearch for analysing I have my logstash config file something like this:

file{
   path => /var/log/rsyslog/**/*.log
}

ISSUE :

Now as number of log files in the directory increase, logstash opens file descriptors (FD) for new files and will not release FDs for already read log files. Since log files are generated per date, once it is read, it is of no use after that since it will not be updated after that date.

I have increased the file openings limit to 65K in /etc/security/limits.conf

Can we make logstash close the handle after some time so that number of file handles opened do not increase too much ??

Answer 1

I think you may have hit this bug: http://github.com/elastic/logstash/issues/1604. Do you have the same symptoms? Exceptions in logs after some time? If you run sudo lsof | grep java | wc -l do you see the descriptors steadily increasing over time? (some of them might close, but some will stay open and their number will increase)