grep multiple patterns single file argument list too long

Question

I am currently searching for multiple patterns in a file. The file is of 90GB in size, I am searching on a particular field(from position 6-17 in each line). I am trying to get all the lines that contain any of a particular list of numbers. The current syntax I am using is:

grep '^.\{6\}0000000012345\|^.\{6\}0000000012543' somelargeFile.txt > outputFile.txt

For small number of patterns this works. For a large number of patterns I get the "Argument list too long" error.

One alternative I have tried is to search for each patters separately (using a for loop over the patterns), but this will require multiple passes over the large data file(57102722 lines) which is not efficient.

From what I understand about the "Argument list too long" error, it is related to bash cmds in general and not specific to grep. Is there any setting that can be used to get around this error? Or alternatively, any ideas as to how to do this using awk or sed or another tool?

Thank you!

Answer 1

You can avoid the problem by putting the patterns in a file, and using the -f command line option to grep.

The most convenient is to put each alternative in a separate line of the file:

patterns.txt

^.\{6\}0000000012345
^.\{6\}0000000012543

invocation

grep -f patterns.txt somelargeFile.txt > outputFile.txt

Answer 2

Try using alternation operator.

grep '^.\{6\}0000000012\(345\|543\)'