Apps aren't loaded yet exception occurs when using multi-processing in Django

Question

I'm doing a Django project and try to improve computing speed in backend.

The task is something like a CPU-bound conversion process

Here's my environment

• Python 3.6.1
• Django 1.10
• PostgreSQL 9.6

And I stuck with following errors when I try to parallel a computing API by python multi-processing library.

  File "D:\\project\apps\converter\models\convert_manager.py", line 1, in <module>
    from apps.conversion.models import Conversion
  File "D:\\project\apps\conversion\models.py", line 5, in <module>
    class Conversion(models.Model):
  File "C:\\virtenv\lib\site-packages\django\db\models\base.py", line 105, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\\virtenv\ib\site-packages\django\apps\registry.py", line 237, in get_containing_app_config
    self.check_apps_ready()
  File "C:\\lib\site-packages\django\apps\registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")

look like each process import Conversion model and Conversion model is like

from django.db import models


    Conversion(model.Model):

       conversion_name = models.CharField(max_length=63)
       conversion_user = models.CharField(max_length=31)
       conversion_description = models.TextField(blank=True)
       ...

Below is my sample function which I want to parallel, each iteration is independent but will access or insert data into SQL.

Class ConversionJob():
     ...

    def run(self, p_list):
        list_merge_result = []
        for p in p_list:
            list_result = self.Couputing_api(p)
            list_merge_result.extend(list_result)

and I'm try to do is

from multiprocessing import Pool


 Class ConversionJob():
         ...
        def run(self, p_list):
            list_merge_result = []

            p = Pool(process=4)
            list_result = p.map(self.couputing_api, p_list)
            list_merge_result.extend(list_result)

In computing_api(), it'll try to get current conversion's info which has completed and save into SQL before this api call, but this caused the error.

My question is

• Why import Conversion model will caused Apps aren't loaded yet errors, I had google lots of article but not actually solve my problems.

• I can see each Process SpawnPoolWorker-x generated and try to boot django server again(why?), each worker will stop at same errors.

• computing API will try to access sql , I haven't think about how to deal with this work. (share db connections or create new connection in each process)

Answer 1

For others that might stumble upon this in future:

If you encounter this issue while running Python 3.8 and trying to use multiprocessing package, chances are that it is due to the sub processed are 'spawned' instead of 'forked'. This is a change with Python 3.8 on Mac OS where the default process start method is changed from 'fork' to 'spawn'. This is a known issue with Django.

To get around it:

import multiprocessing as mp
mp.set_start_method('fork')

Answer 2

This post can solve the problem.

Django upgrading to 1.9 error “AppRegistryNotReady: Apps aren't loaded yet.”

I had found this answer before, but not actually solve my problems at that time.

After I repeated test, I have to add these codes before import another model, otherwise, child-process will booting failed and give the error.

import django
django.setup()
from another.app import models