*nix: perform set union/intersection/difference of lists

Question

I sometimes need to compare two text files. Obviously, diff shows the differences, it also hides the similarities, which is kind of the point.

Suppose I want to do other comparisons on these files: set union, intersection, and subtraction, treating each line as an element in the set.

Are there similarly simple common utilities or one-liners which can do this?

---

Examples:

a.txt

john
mary

b.txt

adam
john

---

$> set_union a.txt b.txt
john
mary
adam

---

$> set_intersection a.txt b.txt
john

---

$> set_difference a.txt b.txt
mary

Answer 1

Union: sort -u files...

Intersection: sort files... | uniq -d

Overall difference (elements which are just in one of the files):
sort files... | uniq -u

Mathematical difference (elements only once in one of the files):
sort files... | uinq -u | sort - <(sort -u fileX ) | uniq -d

The first two commands get me all unique elements. Then we merge this with the file we're interested in. Command breakdown for sort - <(sort -u fileX ):

The - will process stdin (i.e. the list of all unique elements).

<(...) runs a command, writes the output in a temporary file and passes the path to the file to the command.

So this gives is a mix of all unique elements plus all unique elements in fileX. The duplicates are then the unique elements which are only in fileX.

Answer 2

If you want to get the common lines between two files, you can use the comm utility.

A.txt :

A
B
C

B.txt

A
B
D

and then, using comm will give you :

$ comm <(sort A.txt) <(sort B.txt)
        A
        B
C
    D

In the first column, you have what is in the first file and not in the second.

In the second column, you have what is in the second file and not in the first.

In the third column, you have what is in the both files.