Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

grep multiple patterns single file argument list too long

I am currently searching for multiple patterns in a file. The file is of 90GB in size, I am searching on a particular field(from position 6-17 in each line). I am trying to get all the lines that contain any of a particular list of numbers. The current syntax I am using is:

grep '^.\{6\}0000000012345\|^.\{6\}0000000012543' somelargeFile.txt > outputFile.txt

For small number of patterns this works. For a large number of patterns I get the "Argument list too long" error.

One alternative I have tried is to search for each patters separately (using a for loop over the patterns), but this will require multiple passes over the large data file(57102722 lines) which is not efficient.

From what I understand about the "Argument list too long" error, it is related to bash cmds in general and not specific to grep. Is there any setting that can be used to get around this error? Or alternatively, any ideas as to how to do this using awk or sed or another tool?

Thank you!

like image 849
reddy Avatar asked Mar 16 '23 09:03

reddy


2 Answers

You can avoid the problem by putting the patterns in a file, and using the -f command line option to grep.

The most convenient is to put each alternative in a separate line of the file:

patterns.txt

^.\{6\}0000000012345
^.\{6\}0000000012543

invocation

grep -f patterns.txt somelargeFile.txt > outputFile.txt
like image 113
rici Avatar answered Apr 06 '23 13:04

rici


Try using alternation operator.

grep '^.\{6\}0000000012\(345\|543\)'
like image 35
Avinash Raj Avatar answered Apr 06 '23 11:04

Avinash Raj