Crontab No module named Pandas

Question

For some reason, Cron won't process this and keeps telling me that pandas is not installed (it is whenever I normally run my code)

I'm getting this mail:

  Subject: Cron <user@Justins-MBP-4> PYTHONPATH=/Users/user/Library/Python/3.6/lib/python/site-packages python  ~/downloads/random/milbtrans.command
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=user>
X-Cron-Env: <USER=user>
X-Cron-Env: <HOME=/Users/user>
Date: Tue,  8 May 2018 11:18:01 -0400 (EDT)

Traceback (most recent call last):
  File "/Users/user/downloads/random/milbtrans.command", line 2, in <module>
    import requests, csv, pandas, openpyxl, datetime, time
ImportError: No module named pandas

Changed Pandas to be fixed but now getting this error:

 Traceback (most recent call last):
  File "/Users/user/downloads/random/milbtrans.command", line 2, in <module>
    import requests, csv, sys, pandas, openpyxl, datetime, time
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/__init__.py", line 19, in <module>
    "Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['numpy']

Any help is appreciated.

Answer 1

You haven't posted your crontab, but I suspect you are not using the correct path to Python 3.6. Your cron error email says the PATH cron is using is /usr/bin and /bin. Your cron command calls just "python". So cron will use its PATH to try and resolve "python".

Is an executable or link to Python 3.6 available in either of those locations?

What do you see if you run: $ /usr/bin/python or $ /bin/python from your own login? I'm guessing that one, the other, or both would start a different version of Python (i.e. Python 2.x.x)

1. Find out exactly where python3 is installed. Example (your results may be different):

$ which python3 /usr/local/bin/python3

2. In crontab, use this same absolute path when you specify the python executable and the path to your script (also using an absolute path).

crontab

0 0 * * * /usr/local/bin/python3 /Users/user/downloads/random/milbtrans.command

I suggest you try it like this first without the PYTHONPATH.

You could also be more elegant and manage environment variables for the cron execution context (i.e. exporting a correct PATH environment variable via the crontab itself or a "wrapper" shell script) which would also solve the problem, but based on what you've shared here I believe this is the simplest way to address your current issue.

Answer 2

I had the same issue on Ubuntu 18.04.

For me the fix was to have crontab call a shell script which then executes the python script. Also had to set HOME and PYTHONPATH environment variables:

crontab:

# Execute shell script and pipe stdout and stderr to a log file
# Which will enable you to see what's going on
* * * * * <your_path>/yourscript.sh  >> <your_path>yourscript_cron.log 2>&1

yourscript.sh:

#!/bin/bash
echo yourscript.sh called: `date`
HOME=<your_home_dir>
PYTHONPATH=<path_to_dist_packages>
cd <path_to_your_python_script>
<python_executable> ./<your_python_script> 2>&1 1>/dev/null

Funny enough, this shell wrapper method is the way I settled on for scheduled tasks on windows in the past as it gives you the best debugging and flexibility possible.