How do I sync data with remote database in case of offline-first applications?

Question

• I am building a "TODO" application which uses Service Workers to cache the request's responses and in case a user is offline, the cached data is displayed to the user.
• The Server exposes an REST-ful endpoint which has POST, PUT, DELETE and GET endpoints exposed for the resources.
• Considering that when the user is offline and submitting a TODO item, I save that to local IndexedDB, but I can't send this POST request for the server since there is no network connection. The same is true for the PUT, DELETE requests where a user updates or deletes an existing TODO item

Questions

• What patterns are in use to sync the pending requests with the REST-ful Server when the connection is back online?

Answer 1

What patterns are in use to sync the pending requests with the REST-ful Server when the connection is back online?

Background Sync API will be suitable for this scenario. It enables web applications to synchronize data in the background. With this, it can defer actions until the user has a reliable connection, ensuring that whatever the user wants to send is actually sent. Even if the user navigates away or closes the browser, the action is performed and you could notify the user if desired.

Since you're saving to IndexDB, you could register for a sync event when the user add, delete or update a TODO item

function addTodo(todo) {
  return addToIndeDB(todo).then(() => {
    // Wait for the scoped service worker registration to get a
    // service worker with an active state
    return navigator.serviceWorker.ready;
  }).then(reg => {
    return reg.sync.register('add-todo');
  }).then(() => {
    console.log('Sync registered!');
  }).catch(() => {
    console.log('Sync registration failed :(');
  });
}

You've registered a sync event of type add-todo which you'll listen for in the service-worker and then when you get this event, you retrieve the data from the IndexDB and do a POST to your Restful API.

self.addEventListener('sync', event => {
if (event.tag == 'add-todo') {
      event.waitUntil(
      getTodo().then(todos => {
        // Post the messages to the server
        return fetch('/add', {
          method: 'POST',
          body: JSON.stringify(todos),
          headers: { 'Content-Type': 'application/json' }
        }).then(() => {
          // Success!
        });
      })
      })
    );
   }
});

This is just an example of how you could achieve it using Background Sync. Note that you'll have to handle conflict resolution on the server.

You could use PouchDB on the client and Couchbase or CouchDB on the server. With PouchDB on the client, you can save data on the client and set it to automatically sync/replicate the data whenever the user is online. When the database synchronizes and there are conflicting changes, CouchDB will detect this and will flag the affected document with the special attribute "_conflicts":true. It determines which one it'll use as the latest revision, and save the others as the previous revision of that record. It does not attempt to merge the conflicting revision. It is up to you to dictate how the merging should be done in your application. It's not so different from Couchbase too. See the links below for more on Conflict Resolution.

• Conflict Management with CouchDB
• Understanding CouchDB Conflict
• Resolving Couchbase Conflict
• Demystifying Conflict Resolution in Couchbase Mobile

I've used pouchDB and couchbase/couchdb/IBM cloudant but I've done that through Hoodie It has user authentication out-of-the box, handles conflict management, and a few more. Think of it like your backend. In your TODO application, Hoodie will be a great fit. I've written something on how to use Hoodie, see links Below:

• How to build offline-smart application with Hoodie
• Introduction to offline data storage and sync with PouchBD and Couchbase

Answer 2

At the moment I can think of two approaches and it depend on what storage options you are using at your backend.

If you are using an RDBMS to backup all data:

The problem with offline first systems in this approach is the possibility of conflict that you may face when posting new data or updating existing data.

As a first measure to avoid conflicts from happening you will have to generate unique IDs for all objects from your clients and in such a way that they remain unique when posted on the server and saved in a data base. For this you can safely rely on UUIDs for generating unique IDs for objects. UUID guarantees uniqueness across systems in a distributed system and depending on what your language of implementation is you will have methods to generate UUIDs without any hassle.

Design your local database such that you can use UUIDs as primary key in your local database. On the server end you can have both, an integer type auto incremented and indexed, primary key and a VARCHAR type to hold the UUIDs. The primary key on server uniquely identifies objects in that table while UUID uniquely identifies records across tables and databases.

So when posting your object to server at the time of syncing you will have to just check if any object with the UDID is already present and take appropriate action from there. When your are fetching objects from the server send both the primary key of the object from your table and the UDID for the objects. This why when you serialise the response in model objects or save them in local database you can tell the objects which have been synced from the ones which haven't as the objects that needs syncing will not have a primary key in your local database, just the UUID.

There may be a case when your server malfunctions and refuses to save data when you are syncing. In this case you can keep an integer variable in your objects that will keep a count of the number of times you have tried syncing it. If this number exceed by a certain value, say 3, you move on to sync the next object. Now what you do with the unsynced objects is up you the policy you have for such objects, as a solution you could discard them or keep them just locally.

If you are not using RDBMS

As an alternate approach, instead of keeping all objects you could keep transactions that each client perform locally to the server. Each client syncs just the transactions and the while fetching you get the current state by working all the transactions from bottom up. This is very similar to what Git uses. It saves changes in your repository in form of transactions like what has been added (or removed) and by whom. The current state of the repository for each user is worked from the transactions. This approach will not result in conflicts but as you can see its a little tricky to develop.