Menu
I would like to concatenate my files. I use
cat *txt > newFile
But I have almost 500000 files and it complains that the
argument list is too long.
Is there an efficient and fast way of merging half a million files?
Thanks
666
Type the cat command followed by the file or files you want to add to the end of an existing file. Then, type two output redirection symbols ( >> ) followed by the name of the existing file you want to add to.
To choose the merge option, click the arrow next to the Merge button and select the desired merge option. Once complete, the files are merged. If there are multiple files you want to merge at once, you can select multiple files by holding down the Ctrl and selecting each file you want to merge.
To concatenate files, we'll use the cat (short for concatenate) command.
To join two or more text files on the Linux command-line, you can use the cat command. The cat (short for “concatenate”) command is one of the most commonly used commands in Linux as well as other UNIX-like operating systems, used to concatenate files and print on the standard output.
If your directory structure is shallow (there are no subdirectories) then you can simply do:
find . -type f -exec cat {} \; > newFile
If you have subdirectories, you can limit the find to the top level, or you might consider putting some of the files in the sub-directories so you don't have this problem!
This is not particularly efficient, and some versions of find allow you to do:
find . -type f -exec cat {} \+ > newFile
for greater efficiency. (Note the backslash before the + is not necessary, but I find it nice for symmetry with the previous example.)
170
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
