Why do imports fail in setuptools entry_point scripts, but not in python interpreter?

Question

I have the following project structure:

project
|-project.py
|-__init__.py
|-setup.py
|-lib
  |-__init__.py
  |-project
    |-__init__.py
    |-tools.py

with project.py:

from project.lib import *

def main():
    print("main")
    tool()

if __name__ == "__main__":
    main()

setup.py:

from setuptools import setup

setup(
    name = "project",
    version="1.0",
    packages = ["project", "project.lib"],
    package_dir = {"project": ".", "project.lib": 'lib/project'},
    entry_points={
        'console_scripts': [
            'project = project.project:main',
        ],
    },
)

tools.py:

def tool():
    print("tool")

If I run

import project.lib.tools
project.lib.tools.tool()

it works as expected, but running the command project fails with

Traceback (most recent call last):
  File "/usr/local/bin/project", line 9, in <module>
    load_entry_point('project==1.0', 'console_scripts', 'project')()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 568, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2720, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2380, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2386, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "build/bdist.linux-x86_64/egg/project/project.py", line 3, in <module>
ImportError: No module named lib

I don't understand why the two interpreters don't have the same default import pathes.

The reason for this setup is that I want to be able to import project.lib.tools, but keep the directory structure with lib/project.

The complete distutils documentation seriously doesn't say a word on how one can import packages after they have been distributed (the difference of setuptools and distutils isn't less misterious - no way of knowing whether the behavior of distutils is extended here or not).

I'm using setuptools 18.4-1 with python 2.7 on Ubuntu 15.10.

If I change the project structure and setup.py as suggested in @AnttiHaapala's answer I'm getting

$ project
Traceback (most recent call last):
  File "/usr/local/bin/project", line 9, in <module>
    load_entry_point('project==1.0', 'console_scripts', 'project')()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 568, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2720, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2380, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2386, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "build/bdist.linux-x86_64/egg/project/project.py", line 3, in <module>
ImportError: No module named lib

Answer 1

Your project structure seems to be b0rken. The standard layout for a distribution is that the setup.py is on the top-level. Your project would then have 1 (top-level) package, namely project, with sub-package project.lib. Thus we get the following directory layout:

Project-0.42/
 +- project/
 |    +- __init__.py
 |    +- lib/
 |    |   +- __init__.py
 |    |   +- tools.py
 |    +- project.py
 +- setup.py

Then in your setup.py you can simply do

from setuptools import find_packages

setup(
    ...
    # remove package_dir, it is unnecessary
    packages=find_packages(),
    ...
)

The package_dir really does not handle top-level + sub-packages simultaneously very well. After that pip remove project so many times that you can be certain you do not have any buggy versions of it installed in the site-packages, and then run python setup.py develop to link the source into site-packages.

---

After that, the problem is that you're using Python 2 with its broken import system which assumes relative imports. In project.py, your import project.lib assumes a relative import by default, and it tries to actually import project.project.lib. As this is not what you want, you should add

from __future__ import absolute_import

at the top of that file. I seriously suggest that you add this (and why not also the division import if you're using / operator anywhere at all), to avoid these pitfalls and to stay Python 3 compatible.