grep a large list against a large file

Question

I am currently trying to grep a large list of ids (~5000) against an even larger csv file (3.000.000 lines).

I want all the csv lines, that contain an id from the id file.

My naive approach was:

cat the_ids.txt | while read line do   cat huge.csv | grep $line >> output_file done 

But this takes forever!

Are there more efficient approaches to this problem?

Answer 1

Try

grep -f the_ids.txt huge.csv 

Additionally, since your patterns seem to be fixed strings, supplying the -F option might speed up grep.

   -F, --fixed-strings           Interpret PATTERN as a  list  of  fixed  strings,  separated  by           newlines,  any  of  which is to be matched.  (-F is specified by           POSIX.)