Python circular import in custom package and __init__.py

Question

I get ImportError: cannot import name 'Result' from partially initialized module 'libs.elastic_search_hunt' (most likely due to a circular import) error when I try to run tests. But I does not see any circular imports in my code.

I have a package, named elastic_search_hunt which contains 3 modules:

1. elastic_query.py
2. elastic_query_result.py
3. search_processor.py

And I also have __init__.py file with following text:

from libs.elastic_search_hunt.elastic_query import Query
from libs.elastic_search_hunt.search_processor import SearchProcessor
from libs.elastic_search_hunt.elastic_query_result import Result

__all__ = ['Query', 'SearchProcessor', 'Result']  # I guess it does not have any effect

elastic_query.py has only external imports.

elastic_query_result.py the same.

search_processor.py has those import:

from . import Query
from . import Result

Then I have a test file, which imports Query class:

from libs.elastic_search_hunt import Query

When I run tests, I get this errors:

test_query.py:2: in <module>
    from libs.elastic_search_hunt import Query
..\src\libs\elastic_search_hunt\__init__.py:2: in <module>
    from libs.elastic_search_hunt.search_processor import SearchProcessor
..\src\libs\elastic_search_hunt\search_processor.py:4: in <module>
    from . import Result
E   ImportError: cannot import name 'Result' from partially initialized module 'libs.elastic_search_hunt' (most likely due to a circular import)

But where is any circular import in my code? I only can assume that when I import Query from tests, it also import search_processor from the __init__.py module which in turn loads Query one more time. But the error is about Result in elastic_query_result module and I see only one import of Result.

When i delete search_processor from __init__.py everything works fine.

I have read a lot of issues about circular imports, but all of them was quite obvious and does not touch the __init__.py. What am I missing?

Answer 1

TL;DR: replace from . import Query with from .elastic_query import Query

Explanation:

When you import something from libs.elastic_search_hunt module it loads __init__.py at first. Since every module executes at first import __init__.py also being executed.

Then Python executes code from __init__.py and at second line

from libs.elastic_search_hunt.search_processor import SearchProcessor

it imports search_processor.py. Since it's first import - file must be executed - therefore all your imports in that file must be executed right now as well:

As you mentioned you have the following imports in your file:

from . import Query
from . import Result

At this point you tell python to load libs.elastic_search_hunt entire module and take Query, Result from it. So Python does.

It makes an attempt to load libs/elastic_search_hunt/__init__.py but wait... it is still not loaded completely. So it must load it, but in order to load it properly it must firstly load search_processor which requires elastic_search_hunt/__init__.py to be loaded.... oh well, there's a loop.

So in order to avoid such behaviour you should explicitly say from which module exactly you wish to load Query and Result, therefore change

from . import Query
from . import Result

to

from .elastic_query import Query
from .elastic_query_result import Result

Example: Failed Example: Success