combine multiple text files and remove duplicates

Question

I have around 350 text files (and each file is around 75MB). I'm trying to combine all the files and remove duplicate entries. The file is in the following format:

ip1,dns1
ip2,dns2
...

I wrote a small shell script to do this

#!/bin/bash
for file in data/*
do
    cat "$file" >> dnsFull
done
sort dnsFull > dnsSorted
uniq dnsSorted dnsOut
rm dnsFull dnsSorted

I'm doing this processing often and was wondering if there is anything I could do to improve the processing next time when I run it. I'm open to any programming language and suggestions. Thanks!

Answer 1

First off, you're not using the full power of cat. The loop can be replaced by just

cat data/* > dnsFull

assuming that file is initially empty.

Then there's all those temporary files that force programs to wait for hard disks (commonly the slowest parts in modern computer systems). Use a pipeline:

cat data/* | sort | uniq > dnsOut

This is still wasteful since sort alone can do what you're using cat and uniq for; the whole script can be replaced by

sort -u data/* > dnsOut

If this is still not fast enough, then realize that sorting takes O(n lg n) time while deduplication can be done in linear time with Awk:

awk '{if (!a[$0]++) print}' data/* > dnsOut