Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Find unique lines

How can I find the unique lines and remove all duplicates from a file? My input file is

1 1 2 3 5 5 7 7 

I would like the result to be:

2 3 

sort file | uniq will not do the job. Will show all values 1 time

like image 604
amprantino Avatar asked Dec 08 '12 14:12

amprantino


People also ask

How do I remove duplicates in a text file?

The uniq command is used to remove duplicate lines from a text file in Linux. By default, this command discards all but the first of adjacent repeated lines, so that no output lines are repeated. Optionally, it can instead only print duplicate lines. For uniq to work, you must first sort the output.


2 Answers

uniq has the option you need:

   -u, --unique           only print unique lines 
$ cat file.txt 1 1 2 3 5 5 7 7 $ uniq -u file.txt 2 3 
like image 191
Lev Levitsky Avatar answered Sep 30 '22 01:09

Lev Levitsky


Use as follows:

sort < filea | uniq > fileb 
like image 44
kasavbere Avatar answered Sep 30 '22 01:09

kasavbere