How to know if an API response has changed?

Question

I'm working with a REST API which is returning the data I want as a JSON response. Effectively, I want to only process new data that hasn't been processed before, but I'm struggling to know the best way to go about this.

One idea I had was to hash the whole JSON response and store it in a variable, then every time I poll the API for new data, I hash it and check it against the hash of the previous call, then if the hash is different I know the data is different. Obviously, this seems like a very inefficient way to know if there is new data, and was wondering how I can do this better.

I'm using the request npm module.

Answer 1

What you can try to do is to implement a set of HTTP headers by yourself. The headers you are looking for are:

• ETag
• If-Modified-Since
• If-Unmodifed-Since
• If-Modified
• If-Match

You'll find basic characteristics of the mentioned headers here. For the purpose of your application it seems that ETag will be the best. Every response from the server should have ETag header. Next time when the same set of data is requested ETag should be included in request. Server decides if data has changed since the last call (based on ETag). If so, 200 is returned along with the new set, otherwise 304.