Is there a proper place for a staging database?

Question

We have been storing our staging database on the production database server with the mindset that it makes sense to be as identical to production as possible.

Lately, some comments have made me question that idea. Since there is some remote chance that I will do something to production by mistake it may make sense to not put both on the same server.

Should my Staging database really live on the same server as my development database and not the same server as production?

Answer 1

Ideally you would want to have a separate staging environment that mirrors your live environment, but doesn't actually exist on it. However, $$$ doesn't always permit this, so the ideal isn't always followed.

This includes (but may not be limited to) the following:

• Web servers
• Database servers
• Application servers

And anything on those machines (physical or virtual) should be isolated in their respective environments, so you shouldn't see staging code on a production server, and similarly you shouldn't see a staging database on a production database server. They should be separate.

Also, if you use a high amount of bandwidth internally you may want to even isolate the networks, to prevent the staging environment bandwidth usage from saturating the production environment's bandwidth.

Answer 2

In my book a staging environment should be independent because it lets you rehearse the roll-out procedures for a new release. If you are on the same box or same virtual machine you aren't getting the "full" experience of library updates and the like.

Personally I like virtual machines because I can pull production back to stage and then update it. This means that my update is very realistic, because all of the edge case data, libraries and such are being reproduced. This is a good thing... I can't count the number of times over the 9 year history of our major product that a library module wasn't included or some update script for the database hit edge cases that weren't detected in development and testing environments.

As far as touching the production environment... I would say never do this if there is an alternative. Update a shared library in staging that also impacts production and you will feel the pain. Update the code and cause your web server to go into a tizzy and you brought (at least part of) your live environment down.

If you have to fake it, I would recommend sharing with the development environments and just realize that updating production make cause unexpected downtime during the update as you validate everything works. We had to do that for the first few years for budgetary reasons and it can work as long as you don't just update production and walk away.

In summary

• Production is sacrosanct: don't share any non-production aspects if you can avoid it.
• Virtual machines are your friend: they let you clone working environments and update them with nearly zero risk (just copy the VM file over any botched update attempts).
• Staging should be isolated from development to avoid overconfidence with your update routine.