How do I run Python script from a subdirectory?

Question

I want to be able to write a line of code like this and have it run smoothly:

python /path/to/python_file.py -arg1 -arg2 -etc

I figured out an easy way to discover all modules and add them to the current Python path, but it still doesn't seem to recognize the .py file, even though it's supposedly in the sys.path. I know the sys.path addition is working because I can perform this in the interpreter just fine:

>>>import ModuleManager # My Python script to discover modules in lower directories
>>>import testModule # Module I want to run from lower directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named testModule

>>>ModuleManager.discoverModules() # Now, I find the modules and add them to path
Discovering Modules...Complete!
Discovered 6 Modules.
>>>import testModule # No error now.
>>>

What I would like to do at this point is be able to go into my terminal and say:

python testModule -arg1 -arg2 -etc

and have it perform how I would expect.

To be clear, I want to write a line of code in ModuleManager.py (from my application root folder) that allows me to access a file named testModule.py which is found in /root/path/to/testModule.py in a way such that I can use arguments as in python testModule.py -arg1 -arg2 -etc. Any suggestions?

Answer 1

How to write python scripts efficiently

There are multiple ways to write Python script (a script, which is supposed to be called from command line).

For flexible use, forget about manipulating PYTHONPATH - such solutions are difficult to maintain and re rather shaky.

Script with Python stdlib imports only

Betting purely on packages and modules, which are part of Python stdlib makes the script easy to run from anywhere.

Extension to this is importing modules and packages, which are globally installed. Globally installing packages is mostly considered bad practice, as it spoils global Python environment. I have globally installed just small set of packages, which I use often: pyyaml, docopt, lxml, jinja2.

Script importing any installed package

Packages and modules, even your own, can be installed by means of setup.py. It took me a while to get used to this, I was initially ignoring setup.py as too complex solution, but later on I used to use it for any home-made package which I need to import in my programs.

Importing packages installed into virtualenv

This is by far my most popular solution now.

You will need your packages having setup.py.

After virtualenv is created, you use pip to install all the packages you need.

With given virtualenv being active, you can than simply use the script, which will see the packages available for import regardless of where you are your script starting from.

Script installed by means of setup.py

Again, this solution seems first too complex "just for simple script", but finally it can become the simplest method.

You have to create simple Python project with your script and setup.py only. The setup.py shall declare the script for installing. There are two methods, either by scripts argument, or by entry_points, later one being more flexible - use that one.

The script is best installed from within virtualenv - after the installation is complete, find location of the script and move it whenever you want to use it. For using the script, there is no need to activate the virtualenv, it is enough to have it in PATH. The script inside includes refernces to Python from given virtualenv and this ensures, it uses it including all installed packages.

Note, that the best solution for having required packages installed for your script is mentioning it in install_requires argument in setup.py, it ensures, they get installed with your script automatically.

Using zc.buildout

zc.buildout used to be my favourite solution until pip and virtualenv became mature solutions.

You have to create buildout.cfg file and there you define all what is needed for your script. this moslty includes having the setup.py in place, but this is not always necessary.

Personally, I found zc.buildout powerfull, but rather difficult to learn and I would recommend using virtualenv solution.

Using other solutions like PyInstaller

There are other solution, which allow turning Python script into single executable file.

PyInstaller looks like good way to go. There might be other similar solutions.

Conclusions

Forget about making your modules and packages importable by manipulating PYTHONPATH, such solutions are quite shaky.

My recommendation is to go for project having setup.py using entry_points to install your script.

Installation into virtualenv seems to be the cleanest method.

Setting up this environment (the project) the first time takes some effort, but after that, you can simply copy and adopt the solution and it will become natural way to go.

Answer 2

sys.path.append(os.getcwd()) work for me. Now you can run commands like ./manage/my_cli_command.py