I want to have a cron job execute a python script using an already existing anaconda python environment called my_env. The only thing I can think to do is have the cron job run a script called my_script.bash
which in turn activates the env and then runs the python script.
#!/bin/bash source activate my_env python ~/my_project/main.py
Trying to execute this script from the command lines doesn't work:
$ sh scripts/my_script.bash scripts/my_script.bash: 9: scripts/my_script.bash: source: not found
What do I need to do to make sure the proper environment is activated. Its ok to explain it to me like I'm 5.
If you have already created a cron job in your panel, you can view it by running crontab -e under your Shell user. If you edit the file to add another cron job below the existing panel one, the panel cron job will continue to function normally in addition to your new edited code.
Cron job syntax Crontabs use the following flags for adding and listing cron jobs. crontab -e : edits crontab entries to add, delete, or edit cron jobs. crontab -l : list all the cron jobs for the current user. crontab -u username -l : list another user's crons.
I recently switched from canopy to Anaconda precisely to get away from having to activate an env in cron jobs. Anaconda makes this very simple, based on the PATH enviornment variable. (I'm using miniconda not the full Anaconds install but I believe anaconda should work the same way)
There are two different approaches, I've tested;
Add a shebang in your python script, main.py
#!/home/users/user_name/miniconda2/envs/my_env/bin/python
Add PATH to the top of your crontab
PATH=/home/users/user_name/miniconda2/envs/my_env/bin
Update:
Jérôme's answer and cbarrick's comments are correct. I just got burned using the above approach in a Conda env which needed pynco,
which needs the full conda environment to find proper the nco
commands, such as ncks, ncrcat.
Solved by running a bash script from cron which calls conda activate first.
After MUCH fiddling I got crontab
to activate my conda environment with conda activate my_env
and run the Python interpreter within that environment.
Note I'm using Ubuntu 18.04.
Background
When the Anaconda installer initializes conda, it appends a snippet at the end of the ~/.bashrc
file. This file is executed each time the user opens bash
interactively. The snippet allows the user to run conda
commands (ie conda activate my_env
) from bash
.
Anaconda installer v2020.02 appended the following conda
snippet in ~/.bashrc
:
# >>> conda initialize >>> # !! Contents within this block are managed by 'conda init' !! __conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" if [ $? -eq 0 ]; then eval "$__conda_setup" else if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then . "/opt/anaconda3/etc/profile.d/conda.sh" else export PATH="/opt/anaconda3/bin:$PATH" fi fi unset __conda_setup # <<< conda initialize <<<
/opt/anaconda3/
to be replaced with the correct reference: usually /home/USERNAME/anaconda3/
.The problem
Sourcing ~/.bashrc
in crontab -e
won't work (at least not on Ubuntu).
Explanation:
~/.bashrc
has the following (or similar) line at the beginning of the file:# If not running interactively, don't do anything [ -z "$PS1" ] && return
~/.bashrc
file in crontab
, the rest of the .bashrc
file will not execute because crontab
is not running interactively (see another post on this topic). Which means that the conda
snippet mentioned above will never get executed by crontab
even if we source ~/.bashrc
. The solution I have found is to copy the conda
snippet to a separate file.
conda
snippet from ~/.bashrc
to ~/.bashrc_conda
Copy the snippet mentioned above to another file, for example ~/.bashrc_conda
.
Ensure that:
crontab -e
insert 2 lines to run bash
instead of sh
and to source ~/.bashrc_conda
Run crontab -e
and add the following 2 lines before the cronjob:
SHELL=/bin/bash BASH_ENV=~/.bashrc_conda
Explanation:
SHELL=/bin/bash
means that crontab
will run the cronjobs via bash
instead of sh
(default). See post.
BASH_ENV=~/.bashrc_conda
sources the conda
snippet to bash
run by chrontab
. See post and post.
crontab -e
insert in the cronjob line conda activate my_env;
before the desired .py
script executionExample of entry for a script that would execute at noon 12:30 each day within the desired conda environment:
30 12 * * * conda activate my_env; python /path/to/script.py
Notice conda activate my_env;
before the command to run the Python interpreter.
If the conda
snippet in .bashrc
gets updated by a conda
update, it will of course not be reflected in the separate .bashrc_conda
file. One may need to check for updates from time to time.
One could also to append ; conda deactivate
at the end of that cronjob, but this may be redundant.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With