Why does zumero_sync need to be called multiple times?

Question

According to the documentation for zumero_sync:

If a large amount of information needs to be pulled from the server, this function may need to be called more than once.

In my Android app that uses Zumero that's no problem; I just keep calling zumero_sync until the return value doesn't start with "0;".

However, now I'm trying to write an admin script that also syncs with my server dbfiles. I'd like to use the sqlite3 shell, and have the script pass the SQL to execute via command line arguments. I need to call zumero_sync in a loop (which SQLite doesn't support) to make sure the db is fully synced. If I had to, I could invoke sqlite3 in a loop (reading its output, looking for "0;"), or even write a C++ app to call the SQLite/Zumero functions natively. But it certainly would be easier if a single zumero_sync was enough.

I guess my real question is: could zumero_sync be changed so it completes the sync before returning? If there are cases where the existing behavior is more useful, maybe there could be a parameter for specifying which mode to use?

Answer 1

I see two basic questions here:

(1) Why does zumero_sync() work the way it does?

(2) Can it work differently?

I'll answer (2) first, since it's easier: Yes, it could work differently. Rather, we could (and probably will, soon, you brought this up) implement an additional function, named something like zumero_sync_complete(), which performs [the guts of] zumero_sync() in a loop and returns after the sync is complete.

We didn't implement zumero_sync_complete() because it doesn't add much value. It's a simple loop, so you can darn well write it yourself. :-)

Er, except in scripting environments which don't support loops. Like the sqlite3 shell.

Answer to (1):

The Zumero sync protocol is designed to give the server the flexibility to return partial results if it wants to do so. And for the sake of reducing load on the server (and increasing its scalability) it often does want to do exactly that.

Given that, one reason to expose this to the client is to increase the client's flexibility as well. As long we're making multiple roundtrips, we might as well give the client an opportunity to do something (like, maybe, update a progress bar) in between them.

Another thing a client might want to do in between loop iterations is handle an error.

Or, in the case of a multithreaded client, it might want to deal with changes that happened on the client while the sync is going on.

Which raises the question of how locking should be managed? Do we hold the sqlite write lock during the entire loop? Or only when absolutely necessary?

Bottom line: A robust app would probably want to implement the loop itself so that it can make its own decisions and retain full control over things.

But, as you observe, the sqlite3 shell doesn't have loops. And it's not an app. And it doesn't have threads. Or progress bars. So it's a use case where a simpler-and-less-powerful form of zumero_sync() would make sense.