Remove huge amount of files

Question

I have 8 millions of files in my /tmp and I need to remove them. This server is also running pretty important app and I can not overload it.

I'm using small php script:

<?php
$dir = "/tmp";
$dh = opendir( $dir);
$i = 0;
while (($file = readdir($dh)) !== false) {
  $file = "$dir/$file";
  if (is_file( $file) && (preg_match("/open/", $file))) {
    unlink( $file);
    #echo $file;
    if (!(++$i % 10000)) {
      echo "$i files removed\n";
    }
  }
}
?>

but it makes my app unreachable, even with: $ ionice -c 3 php ./tmp_files_killer.php $ nice -n 20 php ./tmp_files_killer.php

I've changed my script so it wouldn't read /tmp dir all the time:

$ ls -1 /tmp > tmp_files_list.txt

<?php
$file = "tmp_files_list.txt"; 
$infile = fopen($file, "r"); 

while ( !feof( $infile ) ) { 
  $line = rtrim( fgets( $infile ), "\n\r" ); 
  if ($line != null){ 
    $file = "$dir/$line";
    unlink( $file);
    if (!(++$i % 10000)) {
      echo "$i files removed\n";
    }
#    echo $line + "\n";
  } 
} 
?>

but running this script also slows down my app. Process doesn't load CPU and I have enough of memory.

Guys, how to delete these files?

Answer 1

You could get the script to operate in "chunks", then sleep between each chunk.

In your second version, you could add a sleep() in after the echo, say 30 seconds. If you tune the number of files deleted and the time slept, it should keep the server responsive while still performing adequately.

In future, you should run a cleanup job regularly from cron to stop yourself from getting to this point.