What are the best practices for benchmarking different Java collections?

Question

I have couple interesting Java collections in my hand such as:

1. http://code.google.com/p/guava-libraries/
2. Java 7
3. Java 7 concurrent collections
4. Scala collections
5. Homegrown collections that we have at some company

I wonder what would be the best practices to test these API's, from a performance and scalability perspective, i.e. which one is fastest, most scalable, performant, etc. Should I set with million(s) of random elements and use timer or something else? Just wanted to satisfy my curiosity and see which one would win.

Answer 1

Edit: updated links to point to automatically exported project on github

Check out Caliper. It will be having its 1.0 release this fall, but many people are already using it with good results (by building it from source; sorry).

Glance over some of the ScareText at https://github.com/google/caliper/wiki/JavaMicrobenchmarks, though.

Answer 2

There's a white paper on somebody benchmarking Java collections. I didn't see any source code, though.