Selecting unique lines based on two columns

Question

I've been trying to figure out a way to select lines that are unique based on the values of two columns. For example, here is a sample/example of my file:

chr1    10    12
chr1    10    12
chr1    10    11
chr1    9    12
chr2    15    20

And this is what I want my output to look like:

chr1    10    12
chr1    10    11
chr1    9    12
chr2    15    20

I've tried to use this code because I like how it doesn't mess with the order of my file:

awk -F"\t" '!_[$2]++' SNP_positions.txt > SNP_positions_uniq.txt

However, it only captures unique features of one column. Is there a way to edit this to make it work for two columns?

Answer 1

You can perfectly use an index that uses more than one field for the array elements:

awk -F"\t" '!seen[$2, $3]++' file

In this case we use $2, $3 as index. This way, we will get all different elements of the tuples ($2, $3).

Answer 2

The awk solution provided is great and fast, but I came here myself looking for the sort solution, which is probably slower in almost all cases:

cat file | sort -u -k1,1 -k2,2 -k3,3 -s

I found the answer here: https://stackoverflow.com/a/12546627/778533