I am trying to run a list of tasks (here running airflow but it could be anything really) that require to be executed in a existing Conda environment.
I would like to do these tasks:
- name: activate conda environment
# does not work, just for the sake of understanding
command: source activate my_conda_env
- name: initialize the database
command: airflow initdb
- name: start the web server
command: 'airflow webserver -p {{ airflow_webserver_port }}'
- name: start the scheduler
command: airflow scheduler
Of course, this does not work as each task is independent and the conda environment
activation in the first task is ignored by the following tasks.
I guess the issue would be the same if using a python virtualenv
instead of conda
.
How can I achieve each task being run in the Conda environment?
To activate your Conda environment, type source activate <yourenvironmentname> . Note that conda activate will not work on Discovery with this version. To install a specific package, type conda install -n <yourenvironmentname> [package] . To deactivate the current, active Conda environment, type conda deactivate .
Question 1: do i have to activate the conda env every time i open command prompt? No, you do not have to activate conda every time you open CMD. This is simply a matter of choice for the developer.
Each of your commands will be executed in a different process.
source
command, on the other hand, is used for reading the environment variables into the current process only (and its children), so it will apply only to the activate conda environment
task.
What you can try to do is:
- name: initialize the database
shell: source /full/path/to/conda/activate my_conda_env && airflow initdb
args:
executable: /bin/bash
- name: start the web server
shell: 'source /full/path/to/conda/activate my_conda_env && airflow webserver -p {{ airflow_webserver_port }}'
args:
executable: /bin/bash
- name: start the scheduler
shell: source /full/path/to/conda/activate my_conda_env && airflow scheduler
args:
executable: /bin/bash
Before, check what's the full path to activate
on the target machine with which activate
(you need to do it before any environment is sourced). If Conda was installed in a user's space, you should use the same user for the Ansible connection.
Was looking out for something similar. Found a neater solution than having multiple actions:
- name: Run commands in conda environment
shell: source activate my_conda_env && airflow {{ item }}
with_items:
- initdb
- webserver -p {{ airflow_webserver_port }}
- scheduler
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With