Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python: perform relative import when using __import__?

Tags:

python

import

Here are the files in this test:

main.py
app/
 |- __init__.py
 |- master.py
 |- plugin/
 |-  |- __init__.py
 |-  |- p1.py
 |-  |_ p2.py

The idea is to have a plugin-capable app. New .py or .pyc files can be dropped into plugins that adhere to my API.

I have a master.py file at the app level that contains global variables and functions that any and all plugins may need access to, as well as the app itself. For the purposes of this test, the "app" consists of a test function in app/__init__.py. In practice the app would probably be moved to separate code file(s), but then I'd just use import master in that code file to bring in the reference to master.

Here's the file contents:

main.py:

import app

app.test()
app.test2()

app/__init__.py:

import sys, os

from plugin import p1

def test():
        print "__init__ in app is executing test"
        p1.test()

def test2():
        print "__init__ in app is executing test2"
        scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
        print "The scriptdir is %s" % scriptDir
        sys.path.insert(0,scriptDir)
        m = __import__("p2", globals(), locals(), [], -1)
        m.test()

app/master.py:

myVar = 0

app/plugin/__init__.py:

<empty file>

app/plugin/p1.py:

from .. import master

def test():
    print "test in p1 is running"
    print "from p1: myVar = %d" % master.myVar

app/plugin/p2.py:

from .. import master

def test():
    master.myVar = 2
    print "test in p2 is running"
    print "from p2, myVar: %d" % master.myVar

Since I explicitly import the p1 module, everything works as expected. However, when I use __import__ to import p2, I get the following error:

__init__ in app is executing test
test in p1 is running
from p1: myVar = 0
__init__ in app is executing test2
The scriptdir is ....../python/test1/app/plugin
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    app.test2()
  File "....../python/test1/app/__init__.py", line 17, in test2
    m = __import__("p2", globals(), locals(), [], -1)
  File "....../python/test1/app/plugin/p2.py", line 1, in <module>
    from .. import master
ValueError: Attempted relative import in non-package

Execution proceeds all the way through the test() function and errors out right as test2() tries to execute its __import__ statement, which in turn p2 tries to do a relative import (which does work when p1 is imported explicitly via the import statement, recall)

It's clear that using __import__ is doing something different than using the import statement. The Python docs state that using import simply translates to an __import__ statement internally but there has to be more going on than meets the eye.

Since the app is plugin-based, coding explicit import statements in the main app would of course not be feasible. Using import itself within the

What am I missing here? How can I get Python to behave as expected when manually importing modules using __import__? It seems maybe I'm not fully understanding the idea of relative imports, or that I'm just missing something with respect to where the import is occurring (i.e. inside a function rather than at the root of the code file)

EDIT: I found the following possible, but unsuccessful solutions:

m = __import__("p2",globals(),locals(),"plugin")

(returns the same exact error as above)

m = __import__("plugin",fromlist="p2")

(returns a reference to app.plugin, not to app.plugin.p2)

m = __import__("plugin.p2",globals(),locals())

(returns a reference to app.plugin, not to app.plugin.p2)

import importlib
m = importlib.import_module("plugin.p2")

(returns:)

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    app.test2()
  File "....../python/test1/app/__init__.py", line 20, in test2
    m = importlib.import_module("plugin.p2")
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named plugin.p2
like image 824
fdmillion Avatar asked Jun 27 '13 13:06

fdmillion


People also ask

How do you use relative import in Python?

Relative imports make use of dot notation to specify location. A single dot means that the module or package referenced is in the same directory as the current location. Two dots mean that it is in the parent directory of the current location—that is, the directory above.

What does __ import __ do in Python?

The __import__() in python module helps in getting the code present in another module by either importing the function or code or file using the import in python method. The import in python returns the object or module that we specified while using the import module.

What are the 3 types of import statements in Python?

Good practice is to sort import modules in three groups - standard library imports, related third-party imports, and local imports.

Should I use relative or absolute imports Python?

Note that relative imports are based on the name of the current module. Since the name of the main module is always “main”, modules intended for use as the main module of a Python application must always use absolute imports.


3 Answers

I've had a similar problem.
__import__ only imports submodules if all parent __init__.py files are empty. You should use importlib instead

import importlib

p2 = importlib.import_module('plugin.p2')
like image 134
Remco Haszing Avatar answered Nov 14 '22 21:11

Remco Haszing


Have you tried the following syntax:

How to use python's import function properly __import__()

It worked for me with a similar problem...

like image 33
TocToc Avatar answered Nov 14 '22 21:11

TocToc


I never did find a solution, so I ended up deciding to restructure the program.

What I did was set up the main app as a class. Then, I also changed each plugin into a class. Then, as I load plugins using import, I also instantiate the class inside each plugin which has a predefined name, and pass in the reference to the main app class.

This means that each class can directly read and manipulate variables back in the host class simply by using the reference. It is totally flexible because anything that the host class exports is accessible by all the plugins.

This turns out to be more effective and doesn't depend on relative paths and any of that stuff. It also means one Python interpreter could in theory run multiple instances of the host app simultaneously (on different threads for example) and the plugins will still refer back to the correct host instance.

Here's basically what I did:

main.py:

import os, os.path, sys

class MyApp:

    _plugins = []

    def __init__(self):
        self.myVar = 0

    def loadPlugins(self):
        scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )   
        sys.path.insert(0,scriptDir)
        for plug in os.listdir(scriptDir):
            if (plug[-3:].lower() == ".py"):
                m = __import__(os.path.basename(plug)[:-3])
                self._plugins.append(m.Plugin(self))

    def runTests(self):
        for p in self._plugins:
            p.test()

if (__name__ == "__main__"):
    app = MyApp()
    app.loadPlugins()
    app.runTests()

plugin/p1.py:

class Plugin:

    def __init__(self, host):
        self.host = host

    def test(self):
        print "from p1: myVar = %d" % self.host.myVar

plugin/p2.py:

class Plugin:

    def __init__(self, host):
        self.host = host

    def test(self):
        print "from p2: variable set"
        self.host.myVar = 1
        print "from p2: myVar = %d" % self.host.myVar

There is some room to improve this, for example, validating each imported .py file to see if it's actually a plugin and so on. But this works as expected.

like image 24
fdmillion Avatar answered Nov 14 '22 22:11

fdmillion