Solr 5.1: Solr is creating way too many log files

Question

I'm dealing with a problem where Solr 5.1 is creating way too many log files. Every time Solr is restarted, and periodically throughout the week, Solr creates the following files and I need it to stop:

1. Files of the type solr_gc_xxxxxxxx_xxxx, where the x's stand for the date and some kind of identifying number, respectively. These contain garbage collection information.
2. Files of the type solr_log_xxxxxxxx_xxxx, where the x's stand for the date and some kind of identifying number, respectively. These contain the same kind of information you'd find in solr.log.
3. One file of the type solr-[port]-console.log. It always contains only the following text: WARNING: System properties and/or JVM args set. Consider using --dry-run or --exec

In one week I racked up nearly thirty of files of the type 1 and 2!

Even worse, file types 1 and 2 don't seem to respect my log4j.rootlogger setting and instead are filled with INFO level material.

Here are the relevant parts of my log4j.properties file:

#  Logging level
solr.log=logs
log4j.rootLogger=WARN, file

#- size rotation with log cleanup.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.File=${solr.log}/solr.log
log4j.appender.file.MaxBackupIndex=0

What I want to do is the following:

1. Create only solr.log + one backup file. solr.log should be periodically overwritten.
2. Not create any other log file.

What can I do to accomplish this?

Answer 1

So after some time, I figured out how to fix this.

To recap, Solr kept creating a whole bunch of files with the solr_log* and gc_log* patterns on startup and periodically throughout the day. Eventually I had some pretty serious space issues because of the endless amount of logs Solr likes to create.

Navigate to /path/to/solr/bin and locate the solr script, which runs at startup. Open the file, look for the following, and comment out mv "$SOLR_LOGS_DIR/solr.log" "$SOLR_LOGS_DIR/solr_log_$(date +"%Y%m%d_%H%M")":

# backup the log files before starting
if [ -f "$SOLR_LOGS_DIR/solr.log" ]; then
  if $verbose ; then
    echo "Backing up $SOLR_LOGS_DIR/solr.log"
  fi
  mv "$SOLR_LOGS_DIR/solr.log" "$SOLR_LOGS_DIR/solr_log_$(date +"%Y%m%d_%H%M")"
fi

Or remove it, if you like. You could also try not using the -f flag but here at my shop we like it.

This will retain solr.log, but Solr won't make any more backups. If you want daily backups, I recommend configuring a TimeBasedRollingPolicy or, better yet, a DailyRollingFileAppender in the log4j.properties file, which can be found under /path/to/solr/server/resources.

If you want, you can also comment out the mv line for the Solr garbage collection logs, which will leave you with solr_gc.log only.

If, like me, you have other ways you monitor gc for Solr, then you need to turn off gc logging completely.

In the same directory as the solr script, open solr.in.sh (Mac/Linux only, I think solr.cmd is for Windows users) and comment this line out: # Enable verbose GC logging GC_LOG_OPTS="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails \ -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime".

You will need to restart Solr.