Eventual Consistency and Test Cases

Question

What's the best practice for writing test cases when working with eventually consistent data store like MongoDB?

My current setup is Mongodb with a 3-node Master/Slave/Slave setup with slave-ok set to true. This means that the Master Node is used for write-only and the two slave node is used for read-only.

The time it took for the data to be consistent on the slaves is relatively small, and dependent on the operation and data size. For example, ~3 ms for delete operations and ~200ms for batch insert of 1000 objects.

My goal is to test the operations on my Dao. They may be simple ones like getById, delete, insert, or complicated ones like findByExample. I need to verify that they work correctly, with eventual consistence with in some time-out limit being acceptable.

This is what I have current to test the delete operation, for example:

  @Test
  public void deleteTest() throws InstantiationException,
              IllegalAccessException {
        MyObject obj = new MyObject();
        obj.setName("test object");
        obj.save(obj);
        MyObject found = dao.findById(obj.getId());
        logger.info ("before: " + found);
        Assert.assertEquals(obj, found);

        dao.delete(obj.getId());
        MyObject deleted = null;
        long start = System.nanoTime();
        do {
              //TBD: need to add escape condition/timeout, else may be infinite loop....
              deleted = dao.findById(obj.getId());
              logger.info ("While: " + deleted);
        } while (deleted!=null);
        logger.info("It took " + ((System.nanoTime()-start)/1000000.00) + " ms for delete to be consistent");
        Assert.assertEquals(null, d1);
  } 

Answer 1

A couple of thoughts come to mind

1. In production, if you are ready from the slave, you'll never know if you are getting the most current data. That's the tradeoff of a read slave in MongoDB. My experience has been that under normal working conditions the slave is up to date. If you have to get the most current data, query the master.
2. I would definitely start using mms to track your replica lag. This will tell you how far behind your slaves are so that you can get a feel for how quickly the data will be available
3. As for the original testing question, it depends on your goals. Your DAO should be able to read and write the same whether it is a replica or standalone. You just need to make sure that your application understands that the data it queries might not be the most current data.

Answer 2

For what you're doing, you can rely on the fact that with a replica set, mongo will always write to the master. So I would change the deletion test to something like this:

/*
 * get this from the DAO,
 * or use the instance injected into the DAO, etc.
 */
DBCollection collection = something();
DBCursor itemsRemaining = collection.find(); //find everything
itemsRemaining.setReadPreference(ReadPreference.PRIMARY); //force query to the master
Assert.assertEquals(0, itemsRemaining.count());

Doing the test through the DBCollection directly allows you to force the test query to use the master. I would test that findById(anyOldId) will return null when the item isn't in the collection in a separate test.