Django Custom field's attributes making database queries

Question

I am facing a very weird problem in one of my django projects. In my project I have a custom field class that handles foreign keys, one to one and many 2 many model fields. The class is some thing like the following.

from django import forms


class CustomRelatedField(forms.Field):
     def __init__(self, model, limit=None, multiple=False, create_objects=True, *args, *kwargs):
         self.model = model
         self.limit = limit
         self.multiple = multiple
         self.create_objects = create_objects

         super(CustomRelatedField, self).__init__(*args, **kwargs)

    def clean(self, value):
        """ Calls self.get_objects to get the actual model object instance(s)
            from the given unicode value.
        """
        # Do some value processing here
        return self.get_objects(value)

    def get_objects(self, values):
        """ Returns the model object instances for the given unicode values.
        """

        results = []
        for value in values:
            try:
                obj = self.model.object.get_or_create(name=value)[0]
                results.append(obj)
            except Exception, err:
                # Log the error here.

        return results

    def prepare_value(self, value):
        """ Returns the value to be sent to the UI. The value
            passed to this method is generally the object id or
            a list of object id's (in case it is a many to many object).
            So we need to make a database query to get the object and
            then return the name attribute.
        """

       if self.multiple:
           result  = [obj.name for obj in self.model.filter(pk__in=value)]
       else:
           result = self.model.object.get(pk=value)

       return result

Recently while I was playing with the django-toolbar, I found out one of the pages that has a form with the above mentioned fields was ridiculously making multiple queries for the same objects again and again.

While debugging, I found out the prepare_value method was being called again and again. After some more debugging, I realized the culprit was the template. I have a generic template that I use for forms, It looks something like the following:

{% for field in form %}
   {% if field.is_hidden %}
      <!-- Do something here -->
   {% endif %}

   {% if field.field.required %}
      <!-- Do something here -->
   {% endif %}

   <label>{{ field.label }}</label>
   <div class="form-field">{{ field }}</div>

   {% if field.field.widget.attrs.help_text %}
      <!-- Do something here -->
   {% elif field.errors %}
      <!-- Do something here -->
   {% endif %}
{% endfor %}

In the above code, each if statement calls the field class which calls the prepare_value method which then makes the database queries. Each of the following listed is making a database query, I am totally lost to why this is happening and have no clue about any solutions. Any help, suggestions would be really appreciated. Thanks.

• field.is_hidden
• field.field.required
• field.label
• field.label_tag
• field
• field.field.widget.attrs.help_text
• field.errors

Also, why does this happen with my custom field class only, other fields (FKs, O2Os, M2M's) in the application and the application admin, just make one query, even though they are using a similar template.

Answer 1

Problem is with your prepare_value() method which does explicit queries. .get() does not get cached and always hits the db while iterating on .filter() queryset will evaluate that. This might be causing you multiple queries.

This is not seen in default fields because they do not do any queries in prepare_value().

To resolve this, you can try to cache the value and result. If value hasn't changed, return cached result. Something like:

class CustomRelatedField(forms.Field):
    def __init__(self, model, limit=None, multiple=False, create_objects=True, *args, *kwargs):
       self.cached_result = None
       self.cached_value = None
    ...
    def prepare_value(self, value):
       #check we have cached result
       if value == self.cached_value:
           return self.cached_result

       if self.multiple:
           result  = [obj.name for obj in self.model.filter(pk__in=value)]
       else:
           result = self.model.object.get(pk=value)

       #cache the result and value
       self.cached_result = result
       self.cached_value = value    
       return result

Not sure how good/bad this work around though!