Split Python source into separate directories?

Question

Here are some various Python packages my company "foo.com" uses:

com.foo.bar.web
com.foo.bar.lib
com.foo.zig.web
com.foo.zig.lib
com.foo.zig.lib.lib1
com.foo.zig.lib.lib2

Here's the traditional way to store the source on disk:

pysrc/
  com/
    foo/
      bar/
        web/
        lib/
      zig/
        web/
        lib/
          lib1/
          lib2/

PYTHONPATH=pysrc

But for organizational purposes (different teams, different revision control, etc.), we want to store these as follows:

bar/
  pysrc/
    com/
      foo/
        bar/
          web/
          lib/
zig/
  pysrc/
    com/
      foo/
        zig/
          web/
          lib/
            lib1/
            lib2/

PYTHONPATH=bar/pysrc:zig/pysrc

The question is:

Are there any issues with this second method of organization?

For example, if we import com.foo, where would Python look for the __init__.py?

Would symlinking these directories make sense? e.g.:

pysrc/
  com/
    foo/
      bar/ -> symlink to /bar/pysrc/com/foo/
      zig/ -> symlink to /zig/pysrc/com/foo/

Any general code organizational suggestions are welcome.

Answer 1

Python will go through sys.path in order (which includes PYTHONPATH and then some), looking for a com.foo package in each. The first one it finds, it will use to the exclusion of others, unlike Perl or Java which effectively merges together package namespaces. There are things you can do to __path__ that change this behavior, but "first match wins" is how Python behaves out of the box.

As long as you keep all of com.foo.bar entirely in bar/ and all of com.foo.zig entirely in zig/, you shouldn't have any problems with the second layout.

Answer 2

Reading through the PEP 420 page, it looks like you can add the following __init__.py to the shared packages:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

So your directory structure would be as follows (*-marked __init__.py files have the above code):

myroot/
├── bar
│   └── pysrc
│       └── com
│           ├── ****__init__.py****
│           └── foo
│               ├── ****__init__.py****
│               └── bar
│                   ├── __init__.py
│                   ├── lib
│                   │   ├── __init__.py
│                   │   └── barlib.py
│                   └── web
│                       ├── __init__.py
│                       ├── barweb.py
└── zig
    └── pysrc
        └── com
            ├── ****__init__.py****
            └── foo
                ├── ****__init__.py****
                └── zig
                    ├── __init__.py
                    ├── lib
                    │   ├── __init__.py
                    │   ├── lib1
                    │   │   ├── __init__.py
                    │   │   └── ziblib1.py
                    │   └── lib2
                    │       ├── __init__.py
                    │       └── ziblib2.py
                    └── web
                        ├── __init__.py
                        ├── zigweb.py

Set the python path to point to your com/ directories:

barPath=/myroot/bar/pysrc/
zigPath=/myroot/zig/pysrc/
export PYTHONPATH=$PYTHONPATH:$barPath:$zigPath

To test (I tried on 2.7.14 and 3.6.4):

from com.foo.bar.web.barweb import BarWeb
from com.foo.zig.web.zigweb import ZigWeb
b = BarWeb()
z = ZigWeb()

Not having the __init__.py code yields:

ImportError: No module named zig.web.zigweb