Replicate part of production django database to local or staging

Question

I have a Django website with 3 environments (Local, Staging, Production).

Production contains some data that I don't want my developers to have access to (Personal data and financial data of users).

Doing a DB backup restore is not an option for compliance reason.

However we also have some content pages on this website that we manage with Wagtail CMS.

I am looking for a way to sync production data (only some models but specifically the wagtail pages) back to Staging and Developers local environment when they need to.

Ideally I would have a manage command that I can run in another environment to copy data:

Example: ./manage.py sync_from_prod BlogPost this would find all missing blog post in the local or staging environment and would create them in the database. I cannot find any library doing this for Wagtail or Django doing this.

This seems like a common problem and I am surprised to find no Stackoverflow questions or opensource libraries fixing this problem.

If nothing exist I will probably try to write my own django-model-sync (Found this project, but is 3 year old and compatible until django 1.7 and I am on python3 django 1.11)

In order to manage security, a secret could be used by a dev to access a production API exposing the data (over ssl) for example

Answer 1

You can use command dumpdata for your models you want to copy.

Use in production command ./manage.py dumpdata app_name.model_name > model_name.json. That will save all data of chose db table in file model_name.json. Then use loaddata on local or stage server for this file: ./manage.py loaddata model_name.json.

You can read more about this here: https://the-bosha.ru/2016/06/29/django-delaem-damp-bazy-dannykh-i-vosstanavlivaem-iz-nego-s-dumpdata-i-loaddata/ (Russian instruction only)

Answer 2

I struggled mightily with the dumpdata / loaddata and had to assemble the answer from various sources, so I'll write a short summary of what worked for me in the end:

1. On Prod: ./manage.py dumpdata -e contenttypes -e auth.Permission --output data.json
2. On Dev: flush your database ./manage.py flush, if flushing does not work, see below
3. On Dev: ./manage.py loaddata data.json

If flush does not work due to some data inconsistency on Dev which happened to me:

1. delete the db file
2. delete everything except the initials from the migrations folder rm */migrations/0*.py
3. ./manage.py migrations && ./manage.py migrate

Then you should be able to load the data