Trying to understand JSONField for django postgresql

Question

I'm reading the docs on JSONField, a special postgresql field type. Since I intend to create a custom field that subclasses JSONField, with the added features of being able to convert my Lifts class:

class Lifts(object):
    def __init__(self, series):
        for serie in series:
            if type(serie) != LiftSerie:
                raise TypeError("List passed to constructor should only contain LiftSerie objects")
        self.series = series

class AbstractSerie(object):

    def __init__(self, activity, amount):
        self.activity_name = activity.name
        self.amount = amount

    def pre_json(self):
        """A dict that can easily be turned into json."""
        pre_json = {
            self.activity_name:
                self.amount
        }
        return pre_json

    def __str__(self):
        return str(self.pre_json())

class LiftSerie(AbstractSerie):

    def __init__(self, lift, setlist):
        """ lift should be an instance of LiftActivity.
            setList is a list containing reps for each set
            that has been performed.
        """
        if not (isinstance(setlist, collections.Sequence) and not isinstance(setlist, str)):
            raise TypeError("setlist has to behave as a list and can not be a string.")
        super().__init__(lift, setlist)

I've read here that to_python() and from_db_value() are two methods on the Field class that are involved in loading values from the database and deserializing them. Also, in the docstring of the to_python() method on the Field class, it says that it should be overridden by subclasses. So, I looked in JSONField. Guess what, it doesn't override it. Also, from_db_value() isn't even defined on Field (and not on JOSNField either).

So what is going on here? This is making it very hard to understand how JSONField takes values and turns them into json and stores them in the database, and then the opposite when we query the database.

A summary of my questions:

1. Why isn't to_python() overridden in JSONField?
2. Why isn't from_db_value() overridden in JSONField?
3. Why isn't from_db_value() even defined on Field?
4. How does JSONField go about taking a python dict for example, converting it to a JSON string, and storing it in the database?
5. How does it do the opposite?

Sorry for many questions, but I really want to understand this and the docs are a bit lacking IMO.

Answer 1

For Django database fields, there are three relevant states/representations of the same data: form, python and database. In case of the example HandField, form/database representations are the same string, the python representation is the Hand object instance.

In case of a custom field on top of JSONField, the internal python might be a LiftSerie instance, the form representation a json string, the value sent to the database a json string and the value received from the database a json structure converted by psycopg2 from the string returned by postgres, if that makes sense.

In terms of your questions:

1. The python value is not customized, so the python data type of the field is the same as the expected input. In contrast to the HandField example, where the input could by a string or a Hand instance. In the latter case, the base Field.to_python() implementation, which just returns the input would be enough.

2. Psycopg2 already converts the database value to json, see 5. This is also true for other types like int/IntegerField.

3. from_db_value is not defined in the base Field class, but it is certainly taken into account if it exists. If you look at the implementation of Field.get_db_converters(), from_db_value is added to it if the Field has an attribute named like that.

4. The django.contrib.postgres.JSONField has an optional encoder argument. By default, it uses json.dumps without an encoder to convert a json structure to JSON string.

5. psycopg2 automatically convertes from database types to python types. It's called adaptation. Documentation for JSON adaptation explains how that works and can be customized.

Note that when implementing a custom field, I would suggest writing tests for it during development, especially if the mechanisms are not completely understood. You can get inspiration for such tests in for example django-localflavor.