Merging huge sets (HashSet) in Scala

Question

I have two huge (as in millions of entries) sets (HashSet) that have some (<10%) overlap between them. I need to merge them into one set (I don't care about maintaining the original sets).

Currently, I am adding all items of one set to the other with:

setOne ++= setTwo

This takes several minutes to complete (after several attempts at tweaking hashCode() on the members).

Any ideas how to speed things up?

Answer 1

You can get slightly better performance with Parallel Collections API in Scala 2.9.0+:

setOne.par ++ setTwo

or

(setOne.par /: setTwo)(_ + _)

Answer 2

There are a few things you might wanna try:

• Use the sizeHint method to keep your sets at the expected size.
• Call useSizeMap(true) on it to get better hash table resizing.

It seems to me that the latter option gives better results, though both show improvements on tests here.