Find common lines between two files and also their line number

Question

I want to find common lines between two files(large ones), one with 90 million lines and 1 with 100 thousands and also their line number.

comm -12 file1 file2

gives me the common lines, but I want to know the line number from the individual files as well

Answer 1

This solution works for me on my small test files - I'm not sure how it will perform on a file with 90 million lines.

tab=` printf '\t' `
join -t"$tab" -j2 <( cat -n file1 ) <( cat -n file2 )

This works because cat -n prepends a space-padded number followed by a tab character to each line. The join then finds the common lines looking only at the stuff after the first tab.

After the join is complete, you should see the common lines, each followed by two numbers. The first number is the line number from file1 and the second from file2.

Caveat: This will work if the files don't have tab characters already. If that's not the case, you can use sed to convert the first tab to a 'safe' character.

safe="|"
join -t"$safe" -j2 \
  <( cat -n file1 | sed -e "s:\t:$safe:" ) \
  <( cat -n file2 | sed -e "s:\t:$safe:" )

Also, depending on how join is implemented, you may want to have the smaller file listed in the first process substitution and the larger one in the second. This way the smaller file may all fit in memory and the larger file might be scanned and matching lines selected efficiently. I have no idea if this is the case, but it might be worth a shot.