Has anyone published a detailed comparison between different in-memory RDBMSs? [closed]

Question

There are quite a few independent and not-so-independent studies comparing traditional RDBMSs but I haven't managed to find any good material on in-memory databases. I am primarily interested in ones specialized for OLTP.

So far, I managed to find generic white papers on TimesTen and MySQL Cluster, but I have yet to see a head-to-head comparison. There are other alternatives (e.g. from IBM), but there's even less easily available material.

Answer 1

The information is scattered all over the web, but here's what I found out:

Introduction to database benchmarking

The first thing that you need to do is figure out a standard benchmark to use for comparing your in-memory database choices. A wiki page on Oracle gives an overview of the industry standard database benchmarks. If you can find results for the same industry standard benchmarks for TimesTen and MySql Cluster, you'll be able to compare the two. Finding a detailed comparison of all the in-memory databases is quite rare.

Summary: Since you mentioned OLTP, the industry standard benchmark that you're probably interested in is TPC-E which simulates the OLTP workload of a brokerage firm. Other industry standard benchmarks measure decision support and e-commerce database transactions.

Open source database benchmark kits

The Open Source Development Labs Database Test Suite is the most comprehensive open source implementation of those industry standard benchmarks I mentioned earlier. They have four main suite of tests numbered DBT1, DBT2, DBT3, DBT4. They haven't implemented the TPC-E benchmark, but the Database Test 2 (DBT2) is what you are looking for since it simulates OLTP transactions. Everyone simply refers to running the DBT2 benchmark instead of spelling it out as OSDLB DBT2.

Results

MySql

MySql Cluster Website

MySql published benchmarks

You'll need to register to get the white paper emailed to you, but here is the summary that I took from their site:

Ultimately, MySQL Cluster was able to reach 100,000 transactions per minute in an 8-node configuration. This showed impressive scaling improvements when compared to a 2-node cluster which was able to obtain 26,000 transactions per minute. These performance improvements were realized through the use of new Multi Core Intel Xeon servers based on the Intel Core T micro architecture.

Oracle

Oracle TimesTen Website

Unfortunately the DBT2 does not support TimesTen. However I found a whitepaper that gives detailed benchmarking information for TimesTen in an OLTP workload.

IBM

IBM SolidDB Website

Other Resources

1. In Memory DBMS section of a blog that focuses on databases.
2. Transaction Processing Performance Council website - You can search their database for benchmarks from various hardware vendors using various DBMSes. I was unable to find any in memory DBMS benchmarks however.