How to concatenate huge number of files

Question

I would like to concatenate my files. I use

cat *txt > newFile

But I have almost 500000 files and it complains that the

argument list is too long.

Is there an efficient and fast way of merging half a million files?

Thanks

Answer 1

If your directory structure is shallow (there are no subdirectories) then you can simply do:

find . -type f -exec cat {} \; > newFile

If you have subdirectories, you can limit the find to the top level, or you might consider putting some of the files in the sub-directories so you don't have this problem!

This is not particularly efficient, and some versions of find allow you to do:

find . -type f -exec cat {} \+ > newFile

for greater efficiency. (Note the backslash before the + is not necessary, but I find it nice for symmetry with the previous example.)