RavenDB - Fastest Insert Performance - What is the benchmark?

Question

I'm working on a prototype, using RavenDB, for my company to evaluate. We will have many threads inserting thousands of rows every few seconds, and many threads reading at the same time. I've done my first simple insert test, and before going much further, I want to make sure I'm using the recommended way of getting the best performance for RavenDB inserts.

I believe there is a bulk insert option. I haven't investigated that yet, as I'm not sure if that's necessary. I'm using the .NET API, and my code looks like this at the moment:

Debug.WriteLine("Number of Marker objects: {0}", markerList.Count);

StopwatchLogger.ExecuteAndLogPerformance(() =>
{
  IDocumentSession ravenSession = GetRavenSession();
  markerList.ForEach(marker => ravenSession.Store(marker));
  ravenSession.SaveChanges();
}, "Save Marker data in RavenDB");

The StopwatchLogger simply invokes the action while putting a stopwatch around it:

internal static void ExecuteAndLogPerformance(Action action, string descriptionOfAction)
{
    Stopwatch stopwatch = new Stopwatch();
    stopwatch.Start();

    action();

    stopwatch.Stop();

    Debug.WriteLine("{0} -- Processing time: {1} ms", descriptionOfAction, stopwatch.ElapsedMilliseconds);
}

Here is the output from a few runs. Note, I'm writing to a local instance of RavenDB (build 701). I know performance will be worse over the network, but I'm testing locally first.

One run:
Number of Marker objects: 671
Save Marker data in RavenDB -- Processing time: 1308 ms

Another run:
Number of Marker objects: 670
Save Marker data in RavenDB -- Processing time: 1266 ms

Another run:
Number of Marker objects: 667
Save Marker data in RavenDB -- Processing time: 625 ms

Another run:
Number of Marker objects: 639
Save Marker data in RavenDB -- Processing time: 639 ms

Ha. 639 objects in 639 ms. What are the odds of that? Anyway, that's one insert per millisecond, which would be 1000 every second.

The Marker object/document doesn't have much to it. Here is an example of one that has already been saved:

{
  "ID": 14740009,
  "SubID": "120403041588",
  "ReadTime": "2012-04-03T13:51:45.0000000",
  "CdsLotOpside": "163325",
  "CdsLotBackside": "163325",
  "CdteLotOpside": "167762",
  "CdteLotBackside": "167762",
  "EquipmentID": "VA_B"
}

Is this expected performance?

Is there a better way (best practice) to insert to gain speed?

Are there insert benchmarks available somewhere that I can target?

Answer 1

First, I would rather make sure that the number of items you save in a single batch doesn't get too big. There is no hard limit, however it hurts performance and eventually will crash if the transaction size gets too big. Using a value like 1024 items is safe, but it really depends on the size of your documents.

1000 documents per seconds is much lower than the number that you can actually reach with a single instance of RavenDB. You should do inserts in parallel and you can do some sort of tweaking with config option. For instance, you could increase the values defined by the settings beginning with Raven/Esent/. It is also a good idea (like in sql server) to put the logs and indexes to different hard drives. Depending on your concrete scenario you may also want to temporarily disable indexing while you're doing the inserts.

However, in most cases you don't want to care about that. If you need really high insert performance you can use multiple sharded instances and theoretically get an unlimited number of inserts/per second (just add more instances).