Is it possible for a transactional db.get() to return a stale result if a recent transaction raised a "special" exception?

Question

This is in the appengine transaction docs...

Note: If your application receives an exception when committing a transaction, it does not always mean that the transaction failed. You can receive Timeout, TransactionFailedError, or InternalError exceptions in cases where transactions have been committed and eventually will be applied successfully...

Consider the following scenario

1. I update entity A within a transaction.
2. transaction operation results in the above described special "exception" where the transaction have been committed and eventually will be applied
3. I run db.get(entity_a_key_goes_here) within another transaction right after, or almost at the same time as step 2.
4. If step 3 above returns None, I create that entity by setting the key to entity_a_key_goes_here and db.put() it (Step 3 and this step run in the same transaction).

My Question:

Is it ever possible for the transactional db.get() operation at step 3 above to return a stale value (or not the updated value set on step 1)? Are transactional db.get() operations guaranteed to return the freshest result even if the "weird" transaction exception occurs right before it?

Answer 1

The get is not really "transactional"; if a transaction only contains reads, then committing the transaction doesn't actually do anything. Executing a read in a transaction guarantees just one thing: if the value the read returned is not the most current value any more at the time that any writes in the transaction are applied, the transaction will abort and none of the writes will have happened.

So, the following sequence of events is allowed to happen:

1. Start transaction 1.
2. Inside transaction 1, you read entity A and it returns state A1.
3. Inside transaction 1, you update entity A from state A1 to state A2.
4. When committing transaction 1, it fails with one of the mentioned exceptions.
5. Start transaction 2.
6. Inside transaction 2, you read entity A and it returns state A1.
7. Transaction 1 is applied to the datastore in the background, changing A from state A1 to state A2.
8. End transaction 2 (no commit, as there were no writes).

But, this sequence of events is different:

1. Start transaction 1.
2. Inside transaction 1, you read entity A and it returns state A1.
3. Inside transaction 1, you update entity A from state A1 to state A2.
4. When committing transaction 1, it fails with one of the mentioned exceptions.
5. Start transaction 2.
6. Inside transaction 2, you read entity A and it returns state A1.
7. Inside transaction 2, you update entity A from state A1 to state A3.
8. Transaction 1 is applied to the datastore in the background, changing A from state A1 to state A2.
9. Transaction 2 now will not commit successfully, because the write it's about to apply is based on an outdated version of entity A. It will fail and A will be left in state A2.

So, with your step 4 added to the question, you are in the second sequence of events here. It's possible that the get in step 3 returns None even though the entity does exist, but it's not possible for the subsequent write to succeed in that case: the transaction is out of date and thus cannot commit. The transaction will be retried, and the second time around, the get will return the object written in step 1, which is what you wanted.

So the very short answer is: yes, it can return a stale value, but it's guaranteed that if the result was stale, a subsequent write to that entity in the same transaction will fail, so this should not actually cause a problem.

Answer 2

I think the result of step 3 in this situation will be a stale value. In this case, the "freshest" result may not be the entity from step 1.

My suggestion is to use memcache when retrieving values (falling back to a db.get() call when memcache misses), and to update memcache when updating the entity. Use the entity key as the memcache key. In fact, this is how ndb works automatically.