How to move all the files of a directory into multiple directories with a given number of files?

Question

I have a directory containing more than 27000 images.

I want to split these files into folders each containing around 500 images.

It doesn't matter how they are sorted, I just want to separate them.

Answer 1

A "simple" find / xargs would do:

find -maxdepth 1 -type f -print0 | xargs -r -0 -P0 -n 500 sh -c 'mkdir newdir.$$; mv "$@" newdir.$$/' xx

Explanation:

• find
  • -maxdepth 1 prevents find from recursively traversing any directories, safety, not needed if you know you don't have directories
  • -type f only find files
  • -print0 separate files with null char instead of LF (to handle strange names)
• xargs
  • -r don't run with empty argument list
  • -0 read files separated with null
  • -P0 create as many processes as you need
  • -n 500 run each process with 500 arguments
• sh
  • -c run command line script provided as next argument
  • mkdir newdir.$$ make a new directory ending with the shell process PID
  • mv "$@" newdir.$$/ move the arguments of the script (each of them quoted) to the newly created directory
  • xx name for the command line provided script (See sh manual)

Note that this is not something I would use in production, it's based mostly on the on the fact that $$ (pid) will be different for each process executed by xargs

If you need the files sorted you can trow a sort -z between find an xargs.

If you want more meaningful directory names you can use something like this:

echo 1 >../seq
find -maxdepth 1 -type f -print0 |sort -z | xargs -r -0 -P1 -n 500 sh -c 'read NR <../seq; mkdir newdir.$NR; mv "$@" newdir.$NR/; expr $NR + 1 >../seq' xx

• echo 1 > ../seq write the first directory suffix in a file (make sure it's not in the current directory)
• -P1 tell xargs to run one command at a time to prevent race conditions
• read NR <../seq read the current directory suffix from the file
• expr $NR + 1 >../seq write the next directory suffix for the next run
• sort -z sort the files

Answer 2

The following should work:

dest_base="destination"
src_dir="src/"

filesperdir=500
atfile=0
atdir=0
for file in $src_dir/*; do
    if ((atfile == 0)); then
        dest_dir=$(printf "$dest_base/%0.5d" $atdir)
        [[ -d $dest_dir ]] || mkdir -p $dest_dir
    fi
    mv $file $dest_dir
    ((atfile++))
    if ((atfile >= filesperdir)); then
        atfile=0
        ((atdir++))
    fi
done