Django REST Framework: Setting up prefetching for nested serializers

Question

My Django-powered app with a DRF API is working fine, but I've started to run into performance issues as the database gets populated with actual data. I've done some profiling with Django Debug Toolbar and found that many of my endpoints issue tens to hundreds of queries in the course of returning their data.

I expected this, since I hadn't previously optimized anything with regard to database queries. Now that I'm setting up prefetching, however, I'm having trouble making use of properly prefetched serializer data when that serializer is nested in a different serializer. I've been using this awesome post as a guide for how to think about the different ways to prefetch.

Currently, my ReadingGroup serializer does prefetch properly when I hit the /api/readinggroups/ endpoint. My issue is the /api/userbookstats/ endpoint, which returns all UserBookStats objects. The related serializer, UserBookStatsSerializer, has a nested ReadingGroupSerializer.

The models, serializers, and viewsets are as follows:

models.py

class ReadingGroup(models.model):
    owner = models.ForeignKeyField(settings.AUTH_USER_MODEL)
    users = models.ManyToManyField(settings.AUTH_USER_MODEL)
    book_type = models.ForeignKeyField(BookType)
    ....
    <other group related fields>

   def __str__(self):
     return '%s group: %s' % (self.name, self.book_type)

class UserBookStats(models.Model):
    reading_group = models.ForeignKey(ReadingGroup)
    user = models.ForeignKey(settings.AUTH_USER_MODEL)
    alias = models.CharField()

    total_books_read = models.IntegerField(default=0)
    num_books_owned = models.IntegerField(default=0)
    fastest_read_time = models.IntegerField(default=0)
    average_read_time = models.IntegerField(default=0)

serializers.py

class ReadingGroupSerializer(serializers.ModelSerializer):
    users = UserSerializer(many = True,read_only=True)
    owner = UserSerializer(read_only=True)

    class Meta:
      model = ReadingGroup
      fields = ('url', 'id','owner', 'users')

    @staticmethod
    def setup_eager_loading(queryset):
      #select_related for 'to-one' relationships
      queryset = queryset.select_related('owner')

      #prefetch_related for 'to-many' relationships
      queryset = queryset.prefetch_related('users')

      return queryset

class UserBookStatsSerializer(serializers.HyperlinkedModelSerializer):
    reading_group = ReadingGroupSerializer()
    user = UserSerializer()
    awards = AwardSerializer(source='award_set', many=True)

    class Meta:
      model = UserBookStats
      fields = ('url', 'id', 'alias', 'total_books_read', 'num_books_owned', 
              'average_read_time', 'fastest_read_time', 'awards')

    @staticmethod
    def setup_eager_loading(queryset):
      #select_related for 'to-one' relationships
      queryset = queryset.select_related('user')

      #prefetch_related for 'to-many' relationships
      queryset = queryset.prefetch_related('awards_set')

      #setup prefetching for nested serializers
      groups = Prefetch('reading_group', queryset ReadingGroup.objects.prefetch_related('userbookstats_set'))        
      queryset = queryset.prefetch_related(groups)

      return queryset

views.py

class ReadingGroupViewset(views.ModelViewset):

  def get_queryset(self):
    qs = ReadingGroup.objects.all()
    qs = self.get_serializer_class().setup_eager_loading(qs)
    return qs

class UserBookStatsViewset(views.ModelViewset):

  def get_queryset(self):
    qs = UserBookStats.objects.all()
    qs = self.get_serializer_class().setup_eager_loading(qs)
    return qs

I've optimized the prefetching for the ReadingGroup endpoint (I actually posted about eliminating duplicate queries for that endpoint here), and now I'm working on the UserBookStats endpoint.

The issue I'm having is that, with my current setup_eager_loading in the UserBookStatsSerializer, it doesn't appear to use the prefetching set up by the eager loading method in the ReadingGroupSerializer. I'm still a little hazy on the syntax for the Prefetch object - I was inspired by this excellent answer to try that approach.

Obviously the get_queryset method of UserBookStatsViewset doesn't call setup_eager_loading for the ReadingGroup objects, but I'm sure there's a way to accomplish the same prefetching.

Answer 1

prefetch_related() supports prefetching inner relations by using double underscore syntax:

queryset = queryset.prefetch_related('reading_group', 'reading_group__users', 'reading_group__owner') 

I don't think Django REST provides any elegant solutions out of the box for fetching all necessary fields automatically.