Poor Performance for Form Rendering In Django 1.11

Question

My site has a navbar with an advanced search widget (beside the search field), which renders on every page. For each request, a context_processor creates the form so it can be available on that page in the navbar. This form has about a dozen selects with a total of several hundred options. Most of those options are for the currency and country selects, along with about 80 other options. There is an even larger list for "stores" but it is loaded via AJAX so it should not be a factor here.

Performance was fine on Django 1.8, but after upgrading to 1.11 (Pyton 2.7.15), I noticed with NewRelic that over 500 ms are now being used on my most frequent request between the following:

• Render/django/forms/widgets/select_option.html
• Render/django/forms/widgets/select.html
• Render/django/forms/widgets/attrs.html

This seems to be related to 1.11's change to Template-based Widget Rendering (docs), however the only pages I could find talking about related problems were about Django Toolbar which I do not run in production.

I am and already using the Cached Template Loader (which is now default), however I don't know if this helps here. I cannot easily cache this form because as you can see in the code, I set a number of defaults based on the request.

Why is my form suffering so badly from this change? Eliminating two of the bigger selects helps, but surely several hundred options should not take this long to render so it seems to me there is an underlying problem that the quantity is merely exacerbating.

Here are links to to code for the full form and html. (I will include snippets in the question later when we identify the problem, for future readers).

• Search Form
• Search HTML
• Live Site
• Discussion on Mailing List

Update

Following this post, I disabled the largest of these selects and limped by for a year. This week I implemented a workaround so I could re-enable these options without paying the enormous cost. I now cache the template fragment of the form, pass the selected form search options to the front end, and set them with JavaScript.

Answer 1

If you have truly isolated the widget rendering as the performance bottleneck, you can make your own widget for this with a different template.

class OptimizedSelectWidget(forms.Select):
    template_name = "widget_templates/optimized_select.html"

class MyForm(forms.Form):
    field = forms.ChoiceField(choices=XXX, widget=OptimizedSelectWidget)

The simpler you can make the template for OptimizedSelectWidget, the faster this will render. The following is a general purpose (i.e. maximally complex) template for the select widget that supports the full functionality of the select widget. I got this by taking the Django 2.2 dropdown template, and in-lining all of the children templates for options.

<select name="{{ widget.name }}"{% include "django/forms/widgets/attrs.html" %}>{% for group_name, group_choices, group_index in widget.optgroups %}{% if group_name %}
  <optgroup label="{{ group_name }}">{% endif %}{% for option in group_choices %}
  <option value="{{ option.value|stringformat:'s' }}" {% for name, value in option.attrs.items %}{% if value is not False %} {{ name }}{% if value is not True %}="{{ value|stringformat:'s' }}"{% endif %}{% endif %}{% endfor %}>{{ option.label }}</option>{% endfor %}{% if group_name %}
  </optgroup>{% endif %}{% endfor %}
</select>

I've tried this on a page where dropdown rendering was a performance issue and did get some speedup, but it was in the 20% range, not the 5x range. My theory is that template caching is already doing a lot, so the templates themselves are not that high in overhead. That said, this is on Django 2.2, which may have introduced some substantial performance improvements over 1.11.

The solution to this problem I ended up going with (the one that got me the 5x speedup) was to use Angular to move the rendering the dropdown options to the browser. Frontend frameworks like Angular or React can be used just for rendering small parts of the page; you don't have to redo your whole frontend to be able to do this.