Bamboo build-dir excessive space can it be cleaned up with a cron job?

Question

We use Bamboo CI. There are multiple bamboo local agents and parallel builds across many plans. The build-dir in bamboo-home is many hundreds of gigabytes, and analysis shows that it just continually grows as new feature branches are added. Plans seem to be duplicated in each local agent directory, and also directly in build-dir.

Unlike expiring artifacts, Bamboo does not seem to clean this up by itself. For example, if a local agent is removed then the local agents build directory sits there forever taking up a significant amount of space.

Plans can be set to clean up at the end of a build, however this impacts problem analysis in the event of needing to do a post-mortem on the build.

Due to the directory running out of space I have just added a daily cron task to periodically remove files and directories that haven't been accessed for more than 21 days. When I first ran this manually I reclaimed 300GB from a 600GB partition. I want to know if others have encountered this same issue, and if it is safe to externally clean the build-dir in the long term. Could it impact bamboo builds? Is there some bamboo option that I have missed that would do this for me?

Searching on the Atlassian site has not been helpful and yields no answers... what are others doing to tame this space hog?

Answer 1

The cron job has been running for a while now without any issues, and it is keeping the space usage under control.

I have reduced the parameter to 15 days.

My crontab looks like this:

# clean up old files from working directory
0 20 * * * find /<path_to>/bamboo-home/xml-data/build-dir/ -depth -not -path *repositories-cache* -atime +15 -delete

# clean up old backups every Sunday
0 21 * * 0 find /<path_to>/bamboo-home/backups -type f -mtime +28 -delete

# remove any old logs from install directory after 15 days
0 22 * * * find /<path_to>/bamboo/logs/ -type f -mtime +15 -delete

# quick and dirty truncate catalina.out to stop it growing too large (or better still use logrotate) 
0 23 * * * cat /dev/null > /<path_to>/bamboo/logs/catalina.out

I hope this is useful for others trying to tame bamboo's diskspace usage. The first job is the important one, the last three are just housekeeping.

N.B. logrotate is not used on catalina.out due to unique circumstances in my companies outsourced linux environment. I would generally recommend logrotate if possible rather than my quick and dirty truncate method - see answer by Jon V.

Answer 2

While the cron idea works well - the thing that I've also done in the past with Bamboo is to "Clean working directory after each build" options. Basically, for any given job, there's a config option that will clean up the appropriate build-dir/<build_plan_job> directory for a given plan/job:

Actions -> Configure Plan -> click the Job -> Miscelaneous Tab -> first checkbox

While that makes sure that future build scratch areas are cleaned up, it does not help for already existing and/or old builds. Given the normal git style workflow where you have lots of branches (and each branch creates a specific job ID for it (like PLAN-JOB_WITH_BRANCH_NUMBER-BUILD_NUMBER or similar) that gets old/large fast. I just did a quick check, and we're now cleaning up the build areas for most builds (the large ones at least), but we have over 100Gig of build cruft from branches that have been merged loooong ago.

Thanks for the cron example, though, that should work OK for the future.

Unrelated: the more I use Bamboo, the more I love/hate it.

EDIT: as a general comment, I'd try really hard to work with an SA to get a logrotate rule set up/implemented for the catalina.out - overwriting with /dev/null seems like a really bad idea, unless you're already slurping them up with something like ELK or Splunk.

My /etc/logrotate.d/bamboo_catalina_out looks like (using your paths):

/<path_to>/bamboo/logs/catalina.out {
  create 0660 bamboo bamboo
  compress
  copytruncate
  missingok
  rotate 10
  size 100M
}

Finally - is there a reason why you have both the third and fourth cron scripts?