/usr/bin/rename: Argument list too long (mass renaming files)

Question

I am trying to mass rename some files by truncating the files where the first space occurs in the file name. I have written a simple script to do this with rename as such:

for i in *.fa; do rename 's/\s.*//' *.fa; done

This works just fine in a test and produces the following results as desired:

$:~/testenv$ ls
NM_000016.5_LastMex1_4_12 23 0 1 KB882088_3062975-30.fa  NM_000016.5_PastMex1_4_12 23 0 1 KB882088_3062975-30.fa
$:~/testenv$ for i in *.fa; do rename 's/\s.*//' *.fa; done
$:~/testenv$ ls
NM_000016.5_LastMex1_4_12  NM_000016.5_PastMex1_4_12

Unfortunately I have to do this on a lot of files, about 6.7 million. This is giving me the following error:

/usr/bin/rename: Argument list too long

I've tried every sort of trick I can think of, even when I use find it can't seem to pick up the files.

$:~/testenv$ ls
NM_000016.5_astMex1_4_12 23 0 1 KB882088_3062975-30.fa   NM_000016.5_PastMex1_4_12 23 0 1 KB882088_3062975-30.fa
NM_000016.5_LastMex1_4_12 23 0 1 KB882088_3062975-30.fa    
$:~/testenv$ find . -maxdepth 1 -type f -exec sh -c 'rename 's/\s.*//' *.fa' _ {} \;
find: `./NM_000016.5_PastMex1_4_12 23 0 1 KB882088_3062975-30.fa': No such file or directory
find: `./NM_000016.5_astMex1_4_12 23 0 1 KB882088_3062975-30.fa': No such file or directory

Any assistance would be greatly appreciated.

Answer 1

You have this bug because you use *.fa not just as the glob to iterate over, but also expand it onto the command line of an individual rename command, where that list of names overruns your operating system's maximum argument vector length. You won't have that issue with the following:

# run rename once per *.fa file, with only one name given each run
for i in *.fa; do rename 's/\s.*//' "$i"; done

...or its more-efficient cousin:

# only tries to rename files that actually have a space in their name
# runs "rename" only as many times as necessary -- not once per file, like the above.
find . -name '*[[:space:]]*.fa' -exec rename 's/\s.*//' {} +

---

That said, the external rename command isn't strictly required for this use case at all.

# Rename all files with a space in their name, then ending in .fa, to truncate at the space
for i in *[[:space:]]*.fa; do
  mv -- "$i" "${i%%[[:space:]]*}"
done

If you want to keep the extension, that may instead be:

for i in *[[:space:]]*.fa; do
  mv -- "$i" "${i%%[[:space:]]*}.fa"
done