Copying millions of files

Question

I've got about 3 million files I need to copy from one folder to another over my company's SAN. What's the best way for me to do this?

Answer 1

If a straight copy is too slow (although a SAN with write-back caching would be about as fast as anything for this type of operation) you could tar the files up into one or more archives and then expand the archives out at the destination. This would slightly reduce the disk thrashing.

At a more clever level, you can do a trick with tar or cpio where you archive the files and write them to stdout which you pipe to another tar/cpio process to unravel them at their destination.

A sample command to do this with tar looks like:

tar cf - * | (cd [destination dir] ; tar xf - )

Some SANs will also directly clone a disk volume.