Menu
I have a directory that contains millions of files spread out in a hierarchy of folders. This directory is stored on a large remote NFS filesystem. I'd like to retrieve the list of these files as fast as possible.
Is it possible to go faster than find . > list.txt? What factors affect speed? I'm using python, but any solution will go as long as it's fast.
629
Try any one of the following commands to see recursive directory listing: ls -R : Use the ls command to get recursive directory listing on Linux. find /dir/ -print : Run the find command to see recursive directory listing in Linux. du -a . : Execute the du command to view recursive directory listing on Unix.
Using the ls Command. The ls command lists the directories and files contained in a directory. The ls command with the -lR options displays the list (in long format) of the sub-directories in the current directory recursively.
On linux, this was the fastest for me. Use (bash) globbing and printf like this:
printf "%s\n" directory/**/file
printf "%s\x00" directory/**/filename-with-special-characters | xargs -0 command
Seems to be a lot faster than
find directory -name file
or
ls -1R directory | grep file
or even, surprisingly,
ls directory/**/file
This was a local file system though: x86_64 system, ext4 filesystem on an SSD, in a directory structure of over 600,000 directories with multiple files in them.
159
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
