Running Luigi task from cmd - "No module named tasks"

Question

I am having problems running a Luigi task through the Windows cmd. Here are the facts:

• Running Anaconda installed in C:\ProgramData\Anaconda2 (Python 2.7)

• Anaconda has added its paths to the PATH variable but there is no PYTHONPATH variable

• The task i am trying to run is located in C:\....\tasks.py

• Trying to run it as follows:

  C:\.... luigi --module tasks MyTask --dt 20170316
  ImportError: No module named tasks

I tried creating a PYTHONPATH variable and adding the exact path to the directory containing my tasks.py file but it didn't work. Another problem I am having, which may be related is when I launch the luigi scheduler through cmd using:

luigid

it works fine but whenever I try to start it using:

luigid --background

I get the following error:

No module named pwd

It seems like there is something wrong with my setup overall, any help would be appreciated.

Answer 1

I had experienced same issue and solved it. Module which you want to schedule or execute must be located in any folder listed in in sys.path One of the way to achive this in WINDOWS TERMINAL/CMD/ is to navigate to folder that you have python module and execute command:

set PYTHONPATH=%cd%;%PYTHONPATH%

That command will add temporary your current directory to existing PYTHONPATH. If you do not have PYTHONPATH variable in system just skip part after semicolon.
In the same terminal window issue luigi command.

luigi --module tasks MyTask --local-scheduler

If you still experience issues please add your PYTHONPATH to PATH variable using:

set PATH=%PYTHONPATH%;%PATH%

For me addidng current folder to PYTHONPATH works and it is easy to execute using batch files. Alternatively you can add this variable permanently in Windows.

Answer 2

Take the examples directory in the luigi repository (git clone ... and you have the luigi directory). There you can find a few different examples, among them:

• hello_world.py contains stuff like task_namespace = 'examples' (that's the same as the python module examples in the repository where all these python files are saved):
  • this could be executed using just the luigi command (no need to have the daemon luigid) from the outside of the python module examples as: cd luigi && PYTHONPATH=. luigi --module examples.hello_world examples.HelloWorldTask --local-scheduler
• top_artists.py does not contain any reference to things like task_namespace:
  • this could be run from within the python module examples: cd luigi/examples && PYTHONPATH='.' luigi --module top_artists AggregateArtists --local-scheduler --date-interval 2012-06

This worked for me using miniconda (similar to anaconda) and cygwin, but I think it could work even if you don't use cygwin (maybe powershell or cmd don't allow you to concatenate commands using && but you can always run those commands one after the other).

I am not sure of the reasons/explanations, but to troubleshoot a bit this behaviour you could play with hello_world.py and run it as cd luigi/examples && PYTHONPATH=. luigi --module hello_world HelloWorldTask --local-scheduler (please note that the luigi command is invoked without examples. as the prefix for the command parameters), this will give the following exception:

raise TaskClassNotFoundException(cls._missing_task_msg(name))
luigi.task_register.TaskClassNotFoundException: No task HelloWorldTask. Candidates are: Config,ExternalTask,RangeBase,RangeByMinutes,RangeByMinutesBase,RangeDaily,RangeDailyBase,RangeHourly,RangeHourlyBase,Task,TestNotificationsTask,WrapperTask,batch_email,core,email,examples.HelloWorldTask,execution_summary,retcode,scheduler,sendgrid,smtp,worker

To give some hints to the other issue you have with the daemon, I start it on cygwin with a command like this: luigid &. That ampersand suffix gives you back the command line prompt. To check which PID is associated to the daemon then I still use the same command line prompt on cygwin and I run ps aux | grep luigid. This approach probably will work ONLY on cygwin (because of some bash related internals).

Answer 3

Since I just ran into the issue myself. What helped me is to stick literally to what is given in the documentation:

PYTHONPATH='.' luigi --module top_artists AggregateArtists --local-scheduler --date-interval 2012-06

So, the PYTHONPATH definition had to be in the same command as the luigi command. This helped me.