How to run a cron job with pipenv?

Question

To demonstrate the issue I'm having, I will use this simple function, which lives in /Users/X/Code/python/example/start.py:

def say_hello(name):
   print("Saying hello to {}".format(name))

say_hello("John")

I'm using pipenv to set up my packages and environment. In this folder (next to start.py), I have four other files - Pipfile, Pipfile.lock, .env and log.txt.

When I run pipenv run python start.py, everything works fine and I get an output.

Now I want this script to run every minute, so I set up a cron job using crontab -e and here is what I initially had in there:

* * * * * /usr/local/bin/pipenv run python /Users/X/Code/python/example/start.py >> /Users/X/Code/python/example/log.txt 2>&1

Which was giving me an error dump in that log.txt file:

RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Consult http://click.pocoo.org/python3/for mitigation steps.

This system lists a couple of UTF-8 supporting locales that
you can pick from.  The following suitable locales where
discovered: af_ZA.UTF-8, am_ET.UTF-8, be_BY.UTF-8, bg_BG.UTF-8, ca_ES.UTF-8, cs_CZ.UTF-8, da_DK.UTF-8, de_AT.UTF-8, de_CH.UTF-8, de_DE.UTF-8, el_GR.UTF-8, en_AU.UTF-8....

After a fair amount of googling, I came to conclusion that the environment was not setting the correct locales, so I added in that .env file which looks like this:

LC_ALL=en_GB.UTF-8
LANG=en_GB.UTF-8

Then I spent a few more minutes looking at the same error appear in the log.txt over and over again and then I realised that when I call /usr/local/bin/pipenv, it actually needs the correct Pipfile to load the correct environment. So instead, I changed my cron to this:

* * * * * cd /Users/X/Code/python/example && /usr/local/bin/pipenv run python /Users/X/Code/python/example/start.py >> /Users/X/Code/python/example/log.txt 2>&1

So technically, now it should be cding into the folder and THEN running the pipenv stuff, but I still get the same exact error in my log file.

Any help would be much appreciated.

Answer 1

Try adding /usr/local/bin to your PATH as well. It seems that pipenv may need to call other applications in the path where it is also located. When cron runs jobs, it doesn't have the same environment that you get when you log into a shell.

Probably easier to create a wrapper script to call from cron:

cd /Users/X/Code/python/example 
PATH=/usr/local/bin:$PATH
pipenv run python start.py >> /Users/X/Code/python/example/log.txt 2>&1

Answer 2

I recently had this issue with a python program using pexpect and running via cron. As others mentioned cron doesn't have the same environment ie $PATH as the user entering the cron script data. In my case I am using a redhat environment that also has a software collection which provides python3.6 ( as mentioned in this post redhat-py3).

Anyway my solution to the pipenv - cron conundrum was to use penv for creating the virtualenv with the required dependencies. And then use 'pipenv shell' to get a shell command that could be used for activating the virtualenv by way of cron and bash. If I have already installed my dependencies with pipenv and created a Pipfile , then I can check the virtualenv using

pipenv check

If i already have a virtualenv then I will see the below messages.

 Checking PEP 508 requirements…
 Passed!
 Checking installed package safety…
 All good!

Then I can run the pipenv shell

pipenv shell     

Which should result in the following

Launching subshell in virtual environment…
. /<path>/<to>/<virtualenv>/bin/activate

!!!Notice the period "." on the second line of the expected output. This line can then be placed in my cron script (I recommend a shell script ). Which is then called by cron.

The final script run by cron and borrowed from @Chris Shaw would look like this:

#!/bin/bash  
#or whatever shell you use
cd /Users/X/Code/python/example
. /<path>/<to>/<virtualenv>/bin/activate 
# you should specifiy the python version in the below command
#python2.7 start.py >> /Users/X/Code/python/example/log.txt 2>&1
python3 start.py >> /Users/X/Code/python/example/log.txt 2>&1

This solution is brittle, due to the reliance on a specific virtualenv, rather than the pipenv commands. So if you recreate the virtualenv , it will likely result in a new path. Which will need to be updated in your shell script. But it removes the reliance on pipenv to provide the shell environment for cron.

As an added note, I suggest adding an alert via email or something similar if the cron job starts to fail. Cron fails silently usually, and will happily carry on without doing what you want until you discover the failure.

Answer 3

For me, it has been sufficient to directly run the virtualenv's Python executable, for example:

0 5 * * * /home/user/.local/share/virtualenvs/project-30iXwYpI/bin/python /home/user/project/script.py

As wordtronix has pointed out, you can get the location of your virtualenv by running pipenv and seeing what it prints on the console.