pip install --editable: links to wrong path

Question

I've used setuptools for a while now and, more recently, pip, to create distributions for my project, and that all works fine: commands like "python setup.py sdist", "python setup.py install" work as configured. Now I would like to use pip to install as "editable", to ease testing while I'm doing maintenance on this package. So I tried

cd \
pip install -e .\mypackage

This adds the path:c:\mypackage to C:\python27\Lib\site-packages\easy-install.pth. However, in my case this is wrong because mypackage is structured as follows:

C:\mypackage 
    setup.py 
    src 
        mypackage 
            __init__.py
             ... 
    docs 
    tests

so easy-install.pth should contain c:\mypackage\src, not c:\mypackage. I can manually edit easy-install.pth to add "\src" to the added path, then "import mypackage" succeeds, as it should. The same problem occurs if I run from c:\mypackage the command "python setup.py develop", so the problem is likely at the setuptools level.

The setup.py has:the

setup( 
     ... 
     packages = find_packages('src'), 
     package_dir = {'mypackage': 'src/mypackage'}, 
     ... 
)

(the only other setup parameters are text items like author, version etc, not listed since not relevant to problem).

I'd like to not have to edit the path in easy-install.pth. Looked at the docs, couldn't see anything indicating that putting the package source root in a folder separate from setup.py is a problem. What I am doing wrong?

Answer 1

I found the answer to this. Turns out this is at the distutils level (pip relies on setuptools which relies on distutils). The section 2.1 of "Distributing Python Modules" discusses the use of package_dir parameter, indicating that "[if] you keep all Python source under lib, so that modules in the “root package” (i.e., not in any package at all) are in lib, modules in the foo package are in lib/foo, ", then you should use

setup(
   ...
   packages = ['foo'],
   package_dir = {'': 'lib'},
   ...
)

As you can see from the OP, this is indeed the case for me, so I changed to the following:

setup(
   ...
   packages = ['mypackage'],
   package_dir = {'': 'src'},
   ...
)

and this worked. So the question is then why

   package_dir = {'': 'src'}

works for releases and editable install, while

   package_dir = {'mypackage': 'src/mypackage'}

works for releases but not for editable install.

The answer is that by default, distutils (and hence setuptools and pip) expects the "root" of the distribution to be the folder that has the setup.py: any *.py and package folder to be installed in site-packages should be there; if they are elsewhere, it must be told. This is done by having an entry with key '' in package_dir. Since my original package_dir did not have this, distutils assumed the root of my dist was the folder containing the setup.py, and that is what it pointed to for an editable install. The release install worked fine because the package_dir as I had it said that mypackage/init.py was in src/mypackage, which it was, so for "regular" distribution all worked fine.