Building object models around external data

Question

I want to integrate external data into a Django app. Let's say, for example, I want to work with GitHub issues as if they were formulated as normal models within Django. So underneath these objects, I use the GitHub API to retrieve and store data.

In particular, I also want to be able to reference the GitHub issues from models but not the other way around. I.e., I don't intend to modify or extend the external data directly.

The views would use this abstraction to fetch data, but also to follow the references from "normal objects" to properties of the external data. Simple joins would also be nice to have, but clearly there would be limitations.

Are there any examples of how to achieve this in an idiomatic way?

Ideally, this would be would also be split in a general part that describes the API in general, and a descriptive part of the classes similar to how normal ORM classes are described.

Answer 1

If you want to use Django Model-like interface for your Github Issues, why don't use real Django models? You can, for example, create a method fetch in your model, that will load data from the remote api and save it to your model. That way you won't need to make external requests everywhere in your code, but only when you need it. A minimal example will look like these:

import requests
from django.db import models

from .exceptions import GithubAPIError


class GithubRepo(models.Model):
    api_url = models.URLField()  # e.g. https://api.github.com/repos/octocat/Hello-World


class GithubIssue(models.Model):
    issue_id = models.IntegerField()
    repo = models.ForeignKey(GithubRepo, on_delete=models.CASCADE)

    node_id = models.CharField(max_length=100)
    title = models.CharField(max_length=255, null=True, blank=True)
    body = models.TextField(null=True, blank=True)

    """
    Other fields
    """

    class Meta:
        unique_together = [["issue_id", "repo"]]

    @property
    def url(self):
        return f"{self.repo.api_url}/issues/{self.issue_id}"

    def fetch_data(self):
        response = requests.get(self.url)
        if response.status != 200:
            raise GithubAPIError("Something went wrong")

        data = response.json()

        # populate fields from repsonse
        self.title = data['title']
        self.body = data['body']

    def save(
        self, force_insert=False, force_update=False, using=None, update_fields=None
    ):
        if self.pk is None:  # fetch on first created
            self.fetch_data()
            super(GithubIssue, self).save(
                force_insert, force_update, using, update_fields
            )

You can also write a custom Manager for your model that will fetch data every time you call a create method - GithubIssue.objects.create()

Answer 2

The django way in this case would be to write a custom "db" backend.

This repo looks abandoned but still can lead you to some ideas.