How to generate deltas for deleted data in client server architecture?

Question

My colleagues and I have created a few client server architecture programs that use deltas to sync a client with a server. A problem relating to deleting records seems to be showing up across multiple projects. How can a server delete records locally and still have enough information to generate delete delta information for future clients?

Example 1:

A realtime game uses UDP client-server to sync up entities between games. Only deltas containing the modified game state and previously dropped packet data are transmitted. If the server deletes an entity, it can send out a delete delta telling every client to delete that object. This works unless a packet is dropped. This is ok for normal state data, as the server can identify which data was dropped and retransmit deltas from it, but this would mean that the server could not locally delete the server entity (without doing a full confirm from each client), as no data would exist to generate the delete delta from.

Example 2:

A client wants to sync with a database stored on a server. The server stores a modified date for each record on the server. When the client request a sync it transmits the last date that it updated. The server gathers all records that have been modified since that date and sends them down to the client. This works with deletions only if the server keeps every record ever deleted and uses a flag to indicate deletion. The server cannot safety delete a locally stored record.

In both examples, the only solution I can think of is to hold on to some sort of record containing everything ever deleted, or force an entire sync after a certain date/time. These do not seem like sustainable or graceful models. What are some sustainable methods for solving this kind of problem?

Answer 1

Why would that not seem sustainable? A lot of sync frameworks use this and it's called Tombstoning, afaik. The record is deleted or flagged as deleted and in another table (if it concerns databases) a reference to the deleted item is held. That way systems are able to sync records that are deleted. In fact, it's not just about deleting data. Some might say every update of a record is a deletion as well. Because the old version of the record can never be found again. So instead of sending updates to the datastore, only send inserts, if keeping 'old' data around is important.

In case of UDP and packet loss, you'll have to design your software around it. If a game submits data for some entity that was already deleted, just ignore it or respond properly to it. So many options here.

You might also be looking at event sourcing. It's not the holy grale or anything, but it works differently from regular CRUD. Instead it uses CQRS to seperate the command and querying side and event sourcing is used to create a stream of events. You only store the events. So whenever you want to state of some entity, you read back all the events. It probably always starts with an insert, zero or multiple updates and then some delete statement. You'll have everything in there that you need to know. These events are also send out to other components who create read models of the latest state. That way some UI can read from the read model and be blazing fast, but every state change comes through the event stream, handling business logic for every event.

If you're interested in this, read up on CQRS and Event Sourcing from Greg Young and a lot of other people.