Display duplicate lines in two different files

Question

I have two files and I would like to display the duplicate line. I tried this but it doesn't work :

cat id1.txt | while read id; do grep "$id" id2.txt; done

I am wondering if there are any other way to display the duplicate lines in the file. Both of my 2 files contain list of ids. Thank you.

Answer 1

use awk will save your time.

awk 'FNR==NR{lines[$0]=1;next} $0 in lines' id1.txt id2.txt

#explaination
FNR==NR #check whether the File NR equal to NR, 
#which will only be true for the first file
lines[$0]=1 #put the contents into a dictionary, 
#value is 1, key is the lines of the first file
next #do not do the next commands if FNR==NR
$0 in lines #check whether the line in the second file
# is in the dictionary
#if yes, will print the $0
#acturally, I omitted the {print},
#which is default to print by awk if condition is true

Answer 2

Are the files sorted? Can they be sorted?

If sorted:

comm -12 id1.txt id2.txt

If not sorted but using bash 4.x:

comm -12 <(sort id1.txt) <(sort id2.txt)

There are solutions using temporary files if you don't have bash 4.x and 'process substitution'.

You could also use grep -F:

grep -F -f id1.txt id2.txt

This looks for the words in id1.txt that appear in id2.txt. The only problem here is ensuring that an ID 1 doesn't match every ID containing a 1 somewhere. The -w or -x options available in some versions of grep will work here.