How to offer multiple versions of an API with different database schemas?

Question

In Kevin Goldsmith's 2015 talk about microservices at Spotify (from 15:25 - 17:43), he mentions that when they create a new version of an API they just create a new server, and keep the old server running with the old version for as long as there are still clients calling it (in this case, a smart lamp with Spotify embedded on it).

I am confused about how they would be able to maintain and offer older versions for potentially years, when surely there would be database schema changes during that timeframe?

I can see a few possible solutions, but none of them seem very reasonable:

1. Using the same database across all versions, only ever add new tables, and new nullable fields. Never delete fields, nor rename fields, nor set fields to non-nullable, nor delete tables, nor rename tables.
2. Using a different database per version, keep each version's data separate.
3. Using a different database per version, keep each version's data separate, but write a way to migrate and pass the requests from one version to another, so that each version receives the request with the valid parameters for that version.

Solution 1 sounds like it would induce way too much code smell, with legacy code everywhere (which Kevin, in my opinion, seems to suggest they certainly do not do).

Solution 2 sounds like a nightmare to pull data out of for other services, or for reporting. What if the information about an entity that you want is in another version's database than the one you request?

Solution 3 sounds like more of a nightmare as you would have to write code to migrate a request for your version, to the versions above and below yours. This would mean that you can't just leave the existing (the one currently in production) version as-is when creating a new version, as you would need to add migrations to move the request both forward and backward so that all versions received the correct parameters for the request.

Hopefully I am just missing something simple here, and there is a magic solution to make this problem easier, but I really cannot see how they could accomplish this?

Thanks!

Answer 1

I have no idea how spotify does this internally.

Just at a guess though from the way he was talking about it, I'm not sure these microservices were storing any data at all. I got the feeling that they were essentially a presentation layer on top of something else (perhaps an internal service layer).

If a microservice can have multiple versions active, and it is also the owner of some data then one of two things are probably happening:

• Schema consistancy is applied at the service level, if a change is made to the schema it must be updated in all historic versions. In the case of multiple versions being deployed independantly this would nessecitate a deploy of all versions on a schema change (he didnt talk like this was the case, but this is how I've always done versioning)
• Each version owns an independant copy of the data. To do this you really need to have a pub/sub model in the system and perform all modifications as a result of some kind of message. (to me this seems pretty inefficient unless its very low churn, but perhaps at that scale it might make sense, there is also a problem with how to prime a new version with data)

I dont think running a schema and never changing it would really work long term. Things change and changes are nessacary. One of the biggest benifits of microservies/SOA (IMO) is that change is easier as the domains are small and containted. You can do things like completely change the storage mechanism (perhaps even to a new type of storage) reasonably safely and quickly as the volume of code is low.

More thoughts on microservice versioning in my article, Microservice versioning; How to make breaking changes without breaking stuff