Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Logrotate to clean up date stamped files

I'm currently trying to work out a method of tidying up Oracle Recover log files that are created by Cron...

Currently, our Oracle standby recover process is invoked by Cron every 15mins using the following command:

0,15,30,45 * * * * /data/tier2/scripts/recover_standby.sh SID >> /data/tier2/scripts/logs/recover_standby_SID_`date +\%d\%m\%y`.log 2>&1 

This creates files that look like:

$ ls -l /data/tier2/scripts/logs/ total 0 -rw-r--r-- 1 oracle oinstall 0 Feb  1 23:45 recover_standby_SID_010213.log -rw-r--r-- 1 oracle oinstall 0 Feb  2 23:45 recover_standby_SID_020213.log -rw-r--r-- 1 oracle oinstall 0 Feb  3 23:45 recover_standby_SID_030213.log -rw-r--r-- 1 oracle oinstall 0 Feb  4 23:45 recover_standby_SID_040213.log -rw-r--r-- 1 oracle oinstall 0 Feb  5 23:45 recover_standby_SID_050213.log -rw-r--r-- 1 oracle oinstall 0 Feb  6 23:45 recover_standby_SID_060213.log -rw-r--r-- 1 oracle oinstall 0 Feb  7 23:45 recover_standby_SID_070213.log -rw-r--r-- 1 oracle oinstall 0 Feb  8 23:45 recover_standby_SID_080213.log -rw-r--r-- 1 oracle oinstall 0 Feb  9 23:45 recover_standby_SID_090213.log -rw-r--r-- 1 oracle oinstall 0 Feb 10 23:45 recover_standby_SID_100213.log -rw-r--r-- 1 oracle oinstall 0 Feb 11 23:45 recover_standby_SID_110213.log -rw-r--r-- 1 oracle oinstall 0 Feb 12 23:45 recover_standby_SID_120213.log 

I basically want to delete off files older than x days old, which I thought logrotate would be perfect for...

I've configured logrotate with the following config file:

/data/tier2/scripts/logs/recover_standby_*.log {     daily     dateext     dateformat %d%m%Y     maxage 7     missingok } 

Is there something I'm missing to get the desired outcome?

I guess I could remove the date from the Crontab log file, and then have logrotate rotate that file, however then the date in the log file wont reflect the day the logs were generated... i.e. Recoveries on 010313 would be in file with a date of 020313 due to logrotate firing on 020313 and rotating the file...

Any other ideas? And thank-you in advance for any responses.

Regards

Gavin

like image 277
fatmcgav Avatar asked Feb 13 '13 16:02

fatmcgav


People also ask

Does logrotate delete old logs?

logrotate is a very useful tool that can automate the process of breaking up (or rotating), compressing, and deleting old log files. For example, you can set up logrotate such that the log file /var/log/foo is rotated every 30 days, and logs older than 6 months are deleted.

What is Sharedscripts in logrotate?

The sharedscripts means that the postrotate script will only be run once (after the old logs have been compressed), not once for each log which is rotated. Note that the double quotes around the first filename at the beginning of this section allows logrotate to rotate logs with spaces in the name.

What is Missingok in logrotate?

missingok : If the log file is missing, go on to the next one without issuing an error message. noolddir : Logs are rotated in the same directory the log normally resides in (this overrides the olddir option). daily : Log files are rotated every day.

What is Postrotate in logrotate?

Postrotate. Logrotate runs the postrotate script each time it rotates a log specified in a configuration block. You usually want to use this script to restart an application after the log rotation so that the app can switch to a new log.


2 Answers

Logrotate removes files according to order in lexically sorted list of rotated log file names, and also by file age (using last modification time of the file)

  • rotate is maximal number of rotated files, you may find. If there is higher number of rotated log files, their names are lexically sorted and the lexically smallest ones are removed.

  • maxage defines another criteria for removing rotated log files. Any rotated log file, being older than given number of days is removed. Note, that the date is detected from the file last modification time, not from file name.

  • dateformat allows specific formatting for date in rotated files. Man page notes, that the format shall result in lexically correct sorting.

  • dateyesterday allows using dates in log file names one day back.

To keep given number of days in daily rotated files (e.g. 7), you must set rotate to value of 7 and you may ignore maxage, if your files are created and rotated really every day.

If log creation does not happen for couple of days, a.g. for 14 days, number of rotated log files will be still the same (7).

maxage will improve the situation in "logs not produced" scenarios by always removing too old files. After 7 days of no log production there will be no rotated log files present.

You cannot use dateformat as OP shows, as it is not lexically sortable. Messing up with dateformat would probably result in removing other rotated log files than you really wanted.

Tip: Run logrotate from command line with -d option to perform a dry run: you will see what logrotate would do but doesn't actually do anything. Then perform a manual run using -v (verbose) so that you can confirm that what is done is what you want.

Solution: clean logs created by cron

The concept is:

Let cron to create and update the log files, but make small modification to create files, following logrotate standard file names when using default dateext

/data/tier2/scripts/logs/recover_standby_SID.log-`date +\%Y\%m\%d`.log 

Use logrotate only for removing too old log files

  • aim at not existing log file /data/tier2/scripts/logs/recover_standby_SID.log
  • use missingok to let logrotate cleanup to happen
  • set rotate high enough, to cover number of log files to keep (at least 7, if there will be one "rotated" log file a day, but you can safely set it very high like 9999)
  • set maxage to 7. This will remove files which have last modification time higher than 7 days.
  • dateext is used just to ensure, logrotate searches for older files looking like rotated.

Logrotate configuration file would look like:

data/tier2/scripts/logs/recover_standby_SID.log {     daily     missingok     rotate 9999     maxage 7     dateext } 

Solution: rotate directly by logrotate once a day

I am not sure, how is source recovery standby file created, but I will assume, Oracle or some script of yours is regularly or continually appending to a file /data/tier2/scripts/logs/recover_standby_SID.log

The concept is:

  • rotate the file once a day by logrotate
  • working directly with log file containing recovery data /data/tier2/scripts/logs/recover_standby_SID.log
  • daily will cause rotation once a day (in terms of how cron understands daily)
  • rotate must be set to 7 (or any higher number).
  • maxage set to 7 (days)
  • dateext to use default logrotate date suffix
  • dateyesterday used to cause date suffixes in rotated files being one day back.
  • missingok to clean older files even when no new content to rotate is present.

Logrotate config would look like:

data/tier2/scripts/logs/recover_standby_SID.log {     daily     missingok     rotate 7     maxage 7     dateext     dateyesterday } 

Note, that you may need to play a bit with copytruncate and other similar options which are related to how is the source log file created by external process and how it reacts to the act of rotation.

like image 93
Jan Vlcinsky Avatar answered Oct 04 '22 06:10

Jan Vlcinsky


You can use find command to do that task easily! It will delete all 7 Days old files. Put it in crontab and run nightly basis:

$ cd /data/tier2/scripts/logs/     $ /usr/bin/find . -mtime +7 -name "*.log" -print -delete 

Or Better way

$ /usr/bin/find /data/tier2/scripts/logs/ -mtime +7 -name "*.log" -print -delete; 
like image 36
Satish Avatar answered Oct 04 '22 08:10

Satish