Ubuntu: large syslog and kern.log files

Question

Logging into my Ubuntu machine, I get a warning that I am running out of disk space. Tracing back, I find that it is the syslogs, especially the kern.log(s) that are eating up my 1TB disk.

-rw-r----- 1 syslog adm 240G Feb 25 14:22 kern.log
-rw-r----- 1 syslog adm 516G Feb 21 07:59 kern.log.1
-rw-r----- 1 syslog adm 1.1K Feb 15 07:39 kern.log.2.gz
-rw-r----- 1 syslog adm  19K Feb  7 07:56 kern.log.3.gz
-rw-r----- 1 syslog adm  37K Feb  1 07:45 kern.log.4.gz
-rw-r----- 1 syslog adm  23G Feb 25 14:52 syslog
-rw-r----- 1 syslog adm  25G Feb 25 08:11 syslog.1
-rw-r----- 1 syslog adm 1.6G Feb 24 07:49 syslog.2.gz
-rw-r----- 1 syslog adm 1.7G Feb 23 08:18 syslog.3.gz
-rw-r----- 1 syslog adm 3.4G Feb 22 08:19 syslog.4.gz
-rw-r----- 1 syslog adm 3.6G Feb 21 07:59 syslog.5.gz
-rw-r----- 1 syslog adm 6.9G Feb 20 07:38 syslog.6.gz
-rw-r----- 1 syslog adm 7.3G Feb 19 07:36 syslog.7.gz

From the snippet above, you can easily find that kern.log and kern.log.1 is eating up 80% of my 1TB disk. I can get the space by deleting the files, but I think it won't solve the problem.

Does anyone have an idea on what the issue might be? I saw that you can get the logging level by:

cat /proc/sys/kernel/printk

and I get

4    4    1    7

Answer 1

This is an old question, but neither of the previous two answers are good solutions:

• The accepted answer doesn't explain why the disk problem goes away if you fix the underlying system issue (the answer is logrotate), plus your system may keep writing to the logs and fill up your disk before you can even figure out the underlying issue.
• The other answer removes and disables the logs entirely, which is not a good approach as it ignores the underlying issue. Also, you'll probably want those log files later when you're figuring out other system problems -- disabling syslog makes it more difficult to track down future issues!

Instead, here is a safer method that lets you keep the log files while reclaiming disk space while also stopping the log files from doing this again.

1. Safely clear the logs: after looking at (or backing up) the logs to identify your system's problem, clear them by typing > /var/log/syslog (including the >). You may need to be root user for this, in which case enter sudo su, your password, and then the above command).

• Then restart the syslog service (either systemctl restart syslog or service syslog restart).

2. Then, you can force the logs to rotate and delete automatically if they reach a certain size, using logrotate. In this case you can edit the config with sudo nano /etc/logrotate.d/rsyslog and add one line:

/var/log/syslog
{
    rotate 7
    daily
    maxsize 1G # add this line
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

• This will force your syslog to "rotate" (i.e., create a new log file and archive the previous log file) after either 1 day or when the file becomes 1GB, whichever comes first. Note that rotate 7 means your system will only keep 7 total syslog backups so it can only ever take up 7GB of space
• Note: you can change maxsize, rotate N, and other settings to customize your logs -- use the command man logrotate to see more.

3. While you're at it, you may want to add the same setting in the second part of the file, which governs the behavior of other log files (e.g. kern.log for kernel events, auth.log for authentication events, etc.). This setting will make it so that each of these other log files will only take 4GB in total.:

...
{
    rotate 4
    weekly
    maxsize 1G
...
}

This will allow your system to keep logging events without them filling your disk.

For more, see the manual and a similar question.