Understanding Python Web Application Deployment

Question

I think I don't completely understand the deployment process. Here is what I know:

• when we need to do hot deployment -- meaning that we need to change the code that is live -- we can do it by reloading the modules, but
• imp.reload is a bad idea, and we should restart the application instead of reloading the changed modules
• ideally the running code should be a clone of your code repository, and any time you need to deploy, you just pull the changes

Now, let's say I have multiple instances of wsgi app running behind a reverse proxy like nginx (on ports like 8011, 8012, ...). And, let's also assume that I get 5 requests per second.

Now in this case, how should I update my code in all the running instances of the application.

• If I stop all the instances, then update all of them, then restart them all -- I will certainly lose some requests
• If I update each instance one by one -- then the instances will be in inconsistent states (some will be running old code, and some new) until all of them are updated. Now if a request hits an updated instance, and then a subsequent (and related) request hits an older instance (yet to be updated) -- then I will get wrong results.

Can somebody explain thoroughly how busy applications like this are hot-deployed?

Answer 1

For deployment across several hot instances that are behind a load balancer like nginx I like to do rolling deployments with a tool like Fabric.

1. Fabric connects you to Server 1
2. Shut down the web-server
3. Deploy changes, either by using your VCS or transferring tarball with the new application
4. Start up the web-server
5. GOTO1 and connect to the next server.

That way you're never offline, and it's seamless as nginx knows when a webserver is taken down when it tries to round-robin to it and will move onto the next one instead, and as soon as the node/instance is back up it will be back into production usage.

EDIT:

You can use the ip_hash module in nginx to ensure all requests from one IP Address goes to the same server for the length of the session

This directive causes requests to be distributed between upstreams based on the IP-address of the client. The key for the hash is the class-C network address of the client. This method guarantees that the client request will always be transferred to the same server. But if this server is considered inoperative, then the request of this client will be transferred to another server. This gives a high probability clients will always connect to the same server.

What this means to you, is that once your web-server is updated and a client has connected to the new instance, all connections for that session will continue to be forwarded to the same server.

This does leave you in the situation of

1. Client connects to site, gets served from Server 1
2. Server 1 is updated before client finishes whatever they're doing
3. Client potentially left in a state of limbo?

This scenario begs the question, are you removing things from your API/Site which could potentially leave the client in a state of limbo ? If all you're doing is for example updating UI elements or adding pages etc but not changing any back-end APIs you should not have any problems. If you are removing API functions, you might end up with issues.