NoSQL db performance testing

Question

Let's assume you've got a nosql database - redis, cassandra, mongodb. And you need to check the overall performance for this database - various platforms, operation systems, even programming languages which are used for test. It's not tied to a specific application or schema.

• What tests you want to see? Can you please help me to form requirements?
  • How database operates in cluster?
  • In broken cluster?
  • In cloud env?
  • How it can perform queries when 10k connections opened?
• What tools you will use?
  • Is it something like JMeter->http server->database?
  • Jmeter->tcp app->database?
  • Other?

All material I've found about database performance testing is like testing database as a part of some product (specific scheme, specific env). Have you thought about database performance testing when database is product itself?

Looking forward for you help.

-vova

Answer 1

In NoSQL benchmarks and performance evaluations I've put together a list of the benchmarks that are correct in the sense that they clearly define the purpose of the benchmark and compare similar features (apples-to-apples comparisons); there are way too many benchmarks out there that are failing at at least one of these fundamental requirements of a benchmark. Going through those you'll be able to extract the bits that are interesting for your own benchmark plus learn what tools have been used and get some benchmarking code too.

So far the most generic NoSQL benchmark is YCSB (Yahoo Cloud Servicing Benchmark). Recently the Cubrid blog posted the results of running this benchmark against some of the most popular NoSQL solutions and that might give you an idea of how to interpret results.