Menu
I have a file like this:
80.13.178.2
80.13.178.2
80.13.178.2
80.13.178.2
80.13.178.1
80.13.178.3
80.13.178.3
80.13.178.3
80.13.178.4
80.13.178.4
80.13.178.7
I need to display unique entries for repeated line (similar to uniq -d) but only entries that occur more than just twice (twice being an example so flexibility to define the lower limit.)
Output for this example should be like this when looking for entries with three or more occurrences:
80.13.178.2
80.13.178.3
723
When we sort the file, it groups the duplicate lines, and uniq treats them as duplicates. We’ll use sort on the file, pipe the sorted output into uniq, and then pipe the final output into less. A sorted list of lines appears in less.
The reason you see duplicate lines is because, for uniq to consider a line a duplicate, it must be adjacent to its duplicate, which is where sort comes in. When we sort the file, it groups the duplicate lines, and uniq treats them as duplicates. We’ll use sort on the file, pipe the sorted output into uniq, and then pipe the final output into less.
Find unique lines The -u option would cause uniq to print only unique lines. Quoting from man uniq: Obs: Remember to sort before uniq -u because uniq operates on adjacent lines. So what uniq -u actually does is to print lines that don't have identical neighbor lines, but that doesn't mean they are really unique.
We’ll use the -f (fields) option to tell uniq which fields to ignore. We get the same results we did when we told uniq to skip three characters at the start of each line. By default, uniq is case-sensitive. If the same letter appears capped and in lowercase, uniq considers the lines to be different.
Feed the output from uniq -cd to awk
sort test.file | uniq -cd | awk -v limit=2 '$1 > limit{print $2}'
With pure awk:
awk '{a[$0]++}END{for(i in a){if(a[i] > 2){print i}}}' a.txt
It iterates over the file and counts the occurances of every IP. At the end of the file it outputs every IP which occurs more than 2 times.
32
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
