In practice, how eventual is the "eventual consistency" in HRD?

Question

I am in the process of migrating an application from Master/Slave to HRD. I would like to hear some comments from who already went through the migration.

1. I tried a simple example to just post a new entity without ancestor and redirecting to a page to list all entities from that model. I tried it several times and it was always consistent. Them I put 500 indexed properties and again, always consistent...

2. I was also worried about some claims of a limit of one 1 put() per entity group per second. I put() 30 entities with same ancestor (same HTTP request but put() one by one) and it was basically no difference from puting 30 entities without ancestor. (I am using NDB, could it be doing some kind of optimization?)

I tested this with an empty app without any traffic and I am wondering how much a real traffic would affect the "eventual consistency".

I am aware I can test "eventual consistency" on local development. My question is:

Do I really need to restructure my app to handle eventual consistency?

Or it would be acceptable to leave it the way it is because the eventual consistency is actually consistent in practice for 99%?

Answer 1

If you have a small app then your data probably live on the same part of the same disk and you have one instance. You probably won't notice eventual consistency. As your app grows, you notice it more. Usually it takes milliseconds to reach consistency, but I've seen cases where it takes an hour or more.

Generally, queries is where you notice it most. One way to reduce the impact is to query by keys only and then use ndb.get_multi() to load the entities. Fetching entities by keys ensures that you get the latest version of that entity. It doesn't guarantee that the keys list is strongly consistent, though. So you might get entities that don't match the query conditions, so loop through the entities and skip the ones that don't match.

From what I've noticed, the pain of eventual consistency grows gradually as your app grows. At some point you do need to take it seriously and update the critical areas of your code to handle it.