Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to force Logstash to reparse a file?

Tags:

file

logstash

I installed Logstash to parse apache files. It took me quite q while to get the settings right and I always tried on real logs. I noticed (as the documentation says) that logstash "remembers" where it was in a file. Now my setings are Ok and I would like Logstash to "forget". This seems harder than I though. I already did the following:

  • used: start_position => "beginning"

  • deleted the complete "data" folder from elastissearch (and stopped it first)

  • looked at which files where opened by logstash with lsof -p PID and deleted everything which was promising (in my case /tmp/jffi*.tmp)

Still Logstash does not forget and parse only "fresh" files in the folder where the logs are

Any ideas?

like image 757
Christophe Claude Avatar asked Oct 23 '13 16:10

Christophe Claude


People also ask

How do I refresh Logstash?

Just use --reload-interval or send a SIGHUP to reload the config! Warning: In regards to Andrew's response, sending a SIGINT to the logstash process as of version 2.2 will terminate the process, not reload the configuration.

How do you parse with Logstash?

Logstash receives the logs using input plugins and then uses the filter plugins to parse and transform the data. The parsing and transformation of logs are performed according to the systems present in the output destination. Logstash parses the logging data and forwards only the required fields.

How do I reload a config file?

To reload Config, select the loaded configurations you want to reload and click Reload. You can only reload a configuration that has the status of Loaded. To refresh Configs, click Refresh. Information for all of the configs in the table is redisplayed.

What is the correct format of Logstash file?

The Logstash configuration file is a custom format developed by the Logstash folks using Treetop. The grammar itself is described in the source file grammar. treetop and compiled using Treetop into the custom grammar. rb parser.


1 Answers

By default logstash writes the position is last was on to a logfile which usually resides in $HOME/.sincedb. Logstash can be fooled into believing it never parsed the logfile by specifying /dev/null as sincedb_path.

Here the part of the documentation Input File.

Where to write the since database (keeps track of the current position of monitored log files). Defaults to the value of environment variable "$SINCEDB_PATH" or "$HOME/.sincedb".

Config Example

input {     file {         path => "/tmp/logfile_to_analyse"         start_position => "beginning"         sincedb_path => "/dev/null"     } } 
like image 167
flazzarini Avatar answered Sep 17 '22 14:09

flazzarini