How to benchmark a crypto library?

Question

What are good tests to benchmark a crypto library?

Which unit (time,CPU cycles...) should we use to compare the differents crypto libraries?

Are there any tools, procedures....?

Any Idea, comment is welcome!

Thank you for your inputs!

Answer 1

I assume you mean performance benchmarks. I would say that both time and cycles are valid benchmarks, as some code may execute differently on different architectures (perhaps wildly differently if they're different enough).

If it is extremely important to you, I would do the testing myself. You can use some timer (almost all languages have one) or you can use some profiler (almost all languages have one of these too) to figure out the exact performance for the algorithms you are looking for on your target platform.

If you are looking at one algorithm vs. another one, you can look for data that others have already gathered and that will give you a rough idea. For instance, here are some benchmarks from Crypto++: http://www.cryptopp.com/benchmarks.html

Note that they use MB/Second and Cycles/Byte as metrics. I think those are very good choices.

Answer 2

Some very good answers before me, but keep in mind optimizations are a very good way to leak key material by timing attack (for example see how devastating it can be for AES). If there is any chance an attacker can time your operations you want not the fastest but the most constant time library available (and possibly the most constant power usage available, if there is any chance someone can monitor yours). OpenSSL does a great job of keeping on top of current attacks, can't necessarily say the same things of other libraries.