Populate Django Database with data from API Get request

Question

My project is built around making API requests (getting JSON) and outputting information from those requests. 60-80% of the requests are requesting Static Data that will rarely change. I want to store this static data in a database that way I don't have to spend all of my requests on that static data.

I have set up a Model that will hold all of the information coming out of just one of those requests. I have no idea where to put any code for actually filling out that database with information from my request.

Model

 class Champion(models.Model):
     id = models.IntegerField(primary_key=True)
     name = models.CharField(max_length=100)
     title = models.CharField(max_length=255)
     image = models.CharField(max_length=5000)
     ...

Basically I know I need to make 2 requests, one to get all of my id's and then loop through those id's and make a request that will create a Champion in the database. I have written the code for doing that already, I just have no idea where to put this code or how to access it.

Answer 1

So in other words - you have some remote API and local django-powered site. You don't want to fetch data from remote API in each request, so you want to store it local. In that case, you need to synchronize your data with remote data, and problem is: how to start synchronization.

You have some possibilities, but first you should consider when you want run synchronization:

• daily/hourly/weekly/monthly?
• by hand/on request?
• when data is changed (some method on remote API, checking last modification)?

If you want to synchronize when data changes or after some period of data, you need to answer one more question: how you synchronization should affect requests? If it's okay to handle synchronization during requests or if it's okay if requests called during synchronization are still returning old data?

If you're decided to sync it on request - you can simply put synchronization code inside custom management command or into separate view (authorized access only) that will run synchronization.

If it's okay to do synchronization during normal request, you can check in each request if time from last sync is more than X, if data on remote API was updated or if other conditions were met and just run synchronization before returning response (watch out for race conditions!).

If you don't want to do it during request (and it's okay that requests made when sync is happening are still returning old data), you can use celery or system cron jobs.

Celery can run tasks on demand (for example in request, when you're checking if remote data was changed, you can run asynchronous task to do the synchronization and return response based on old data) or run tasks periodicaly (like cron jobs). For normal cron tasks, you can just use custom management commands, like mentioned above.

Answer 2

Create a python file inside your project directory and include the code below. To run this python script you just need to say python filename.py from your terminal. Cheers!

import os
import django

logger = logging.getLogger(__name__)

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "your_project_name.settings")

django.setup()

from your_app_name import models

for champ in Champion.objects.all():
    try:
        #make your API request here
        #suppose champ_image is what you retrieve from the API
        champ.image = champ_image
        champ.save()
        print "updated:"+champ.name
    except:
        print "could not save:"+champ.name
print "Action complete!"