Python pip and conda package manager

Question

Yes, I am asking same old question in python world. Its about package-managers pip and conda.

Pip works perfectly fine with python packages but get stuck with python packages with c-extensions (like pandas, matplotlib, etc). Thats where conda comes in to to save time in breaking head in setting up environments. However I stumbled upon some environments where conda channels are not having the packages like flask-sqlalchemy, flask-httpauth, etc. So you have to use pip for that. And managing your environment with two different package manager sounds weird to me and may be not advisable.

What is the best practices for managing python dependencies using such package-managers. Need expert advises to feel in the gaps.

Any ideas/help would be appreciated.

I also use Pycharm and it can only recognize conda packages not pip packages. Thats another pain. So I have to fall back to vim (which is fine - I love that) but debugging is difficult or its not as easy as pycharm like IDE.

P.S. I use Linux/Mac for development.

Answer 1

Here is an example of a decent workflow (from this blog):

1. Create a project folder in the ~/repos/ directory on my computer.
2. Create an environment.yml file in the directory. Typically the environment name will be the same as the folder name. At minimum, it will specify the version of Python I want to use; it will often include anaconda as a dependency.
3. Create the conda environment with $ conda env create.
4. Activate the conda environment with $ source activate ENV_NAME.
5. Create a .env file containing the line source activate ENV_NAME. Because I have autoenv installed, this file will be run every time I navigate to the project folder in the Terminal. Therefore, my conda environment will be activated as soon as I navigate to the folder.
6. Run $ git init to make the folder a Git repository. I then run $ git add environment.yml && git commit -m 'initial commit' to add the YAML file to the repository.
7. If I want to push the repository to Github, I use $ git create using Github's hub commands. I then push the master branch with $ git push -u origin master.

It is decent but not perfect. Some of the disadvantages are:

1. Immediate dependencies are listed together with the dependencies of dependencies.
2. It is complicated to properly remove packages.
3. It doesn't work when there are packages installed from alternative sources.

However, it is a good start and most teams, from my experience, don't bother to invent solutions better than this one.

P.S.: During the last year, PyCharm significantly improved its conda support and as of April 2017 it recognizes properly both conda and pip packages.