Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to print only the unique lines in BASH?

Tags:

bash

uniq

How can I print only those lines that appear exactly once in a file? E.g., given this file:

mountain
forest
mountain
eagle

The output would be this, because the line mountain appears twice:

forest
eagle
  • The lines can be sorted, if necessary.
like image 840
Village Avatar asked May 19 '14 14:05

Village


People also ask

Which Unix command is used to output only unique lines in a file?

The uniq command can count and print the number of repeated lines. Just like duplicate lines, we can filter unique lines (non-duplicate lines) as well and can also ignore case sensitivity. We can skip fields and characters before comparing duplicate lines and also consider characters for filtering lines.

How do you display unique values in Unix?

The uniq command in UNIX is a command line utility for reporting or filtering repeated lines in a file. It can remove duplicates, show a count of occurrences, show only repeated lines, ignore certain characters and compare on specific fields.

What is uniq in bash?

uniq command is used to detect the adjacent lines from a file and write the content of the file by filtering the duplicate values or write only the duplicate lines into another file.


3 Answers

Use sort and uniq:

sort inputfile | uniq -u

The -u option would cause uniq to print only unique lines. Quoting from man uniq:

   -u, --unique
          only print unique lines

For your input, it'd produce:

eagle
forest

Obs: Remember to sort before uniq -u because uniq operates on adjacent lines. So what uniq -u actually does is to print lines that don't have identical neighbor lines, but that doesn't mean they are really unique. When you sort, all the identical lines get grouped together and only the lines that are really unique in the file will remain after uniq -u.

like image 149
devnull Avatar answered Oct 24 '22 09:10

devnull


Using awk:

awk '{!seen[$0]++};END{for(i in seen) if(seen[i]==1)print i}' file
eagle
forest
like image 35
anubhava Avatar answered Oct 24 '22 09:10

anubhava


You almost had the answer in your question:

sort filename | uniq -u

like image 8
Oliver Matthews Avatar answered Oct 24 '22 09:10

Oliver Matthews