I'm trying to package my local module with pex and I can't seem to manage to.
I've created a simple project:
→ python --version
Python 2.7.10
→ pex --version
pex 1.1.15
→ tree .
.
├── bla
│ ├── __init__.py
│ └── main.py
└── setup.py
bla/__init__.py
import main
bla/main.py
if __name__ == '__main__':
print 'yo'
Which to me seems like the simplest project possible.
→ pex -v . -o v.pex --disable-cache
Traceback (most recent call last):
File "/Users/Charly/repos/load_tester/venv/bin/pex", line 11, in <module>
sys.exit(main())
File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/bin/pex.py", line 540, in main
pex_builder = build_pex(reqs, options, resolver_options_builder)
File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/bin/pex.py", line 475, in build_pex
resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
File "/Users/Charly/repos/load_tester/venv/lib/python2.7/site-packages/pex/resolvable.py", line 61, in get
raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: .
Also tried to do python setup.py bdist_pex
but this failed as well as it can't find the command.
It really seems like I'm misunderstanding something fundamental but I can't figure out what.
I recently had a bit of a fight with pex
trying to make it include local modules. What I learned is:
setup.py
file for your module(s) in order for this to work, and:This was tricky to figure out for several reasons. From reading the documentation, I was able to infer that the correct command in my case should be something like this:
$ pex . -v -e usersnotifier:main -o usersnotifier.pex
However, when I tried this, I kept getting an error saying:
pex.resolvable.InvalidRequirement: Unknown requirement type: .
A web search for this error turns up—as its first hit—this Github issue, which is still open as I type this. A spent a long while thinking that the above command wasn't working because of this bug. I attempted to downgrade setuptools and made a half-dozen other fruitless attempts to 'fix' the problem before this SO answer hinted at the necessity of supplying a setup.py
file. (That Github issue turned out to be a red herring. The setuptools
bug it mentions has since been fixed, from what I can tell.)
So... I wrote a setup.py
file. At first, I kept getting that error saying Unknown requirement type: .
But then I realized that my setup.py
simply contained a dead-obvious typographical error. The error message emitted by pex in this case was actually quite clear, but it was followed by a large-ish stack trace and the Unknown requirement type: .
message. I just wasn't paying close attention and missed it for longer than I care to admit.
I finally noticed my typo and fixed it, but another flaw in my setup.py
was failing to include my local modules. pex
worked in this case, but the generated file didn't:
$ pex . -v -e usersnotifier:main -o usersnotifier.pex --disable-cache
usersnotifier 0.1: Resolving distributions :: Packaging paho-mqtt
pyinotify 0.9.6
paho-mqtt 1.3.1
pex: Building pex: 2704.3ms
pex: Resolving distributions: 2393.2ms
pex: Packaging usersnotifier: 319.3ms
pex: Packaging pyinotify: 347.4ms
pex: Packaging paho-mqtt: 361.1ms
Saving PEX file to usersnotifier.pex
$ ./usersnotifier.pex
Traceback (most recent call last):
File ".bootstrap/_pex/pex.py", line 367, in execute
File ".bootstrap/_pex/pex.py", line 293, in _wrap_coverage
File ".bootstrap/_pex/pex.py", line 325, in _wrap_profiling
File ".bootstrap/_pex/pex.py", line 410, in _execute
File ".bootstrap/_pex/pex.py", line 468, in execute_entry
File ".bootstrap/_pex/pex.py", line 482, in execute_pkg_resources
File ".bootstrap/pkg_resources/__init__.py", line 2297, in resolve
ImportError: No module named 'usersnotifier'
Here's the bare-bones setup.py
that finally worked for me:
from setuptools import setup
setup(
name='usersnotifier',
version='0.1',
py_modules=['usersnotifier', 'userswatcher'],
install_requires=[
'paho-mqtt>=1.3.1',
'pyinotify>=0.9.6',
],
include_package_data=True,
zip_safe=False
)
The reason it hadn't worked before was that I was accidentally passing the parameter py_module
to setup()
rather than py_modules
(plural). ¯\_(ツ)_/¯
The final hurdle I encountered was mentioned in @cmcginty's answer to this question, namely: unless your module version number changes, pex
will cache/reuse artifacts from the last time you ran it. So, if you fix a problem in your setup.py
and re-run pex
, it won't actually incorporate your changes unless you: a) bump the version number, or b) pass --disable-cache
when invoking pex
.
At the end of the day, the whole thing turned into an exercise in writing a proper setup.py
, and running:
$ pex . -v -e mymodule:main -o mymodule.pex --disable-cache
Here are a few tips I can offer (possibly to a future version of my self):
Use python setup.py sdist
to test your setup.py
file. It's surprisingly easy to screw this up, and there's no point involving pex
until you're sure your package has the right contents. After running python setup.py sdist
, try installing the source package it generates (located in the dist
folder) into a fresh venv and see whether it contains all the files you expect. Only move on to invoking pex
after this is working.
Always pass --disable-cache
to pex
unless you have a good reason not to.
While troubleshooting all of these issues, I discovered that I could run:
$ unzip mymodule.pex
to extract the contents of the PEX file. This can be helpful in resolving any discrepancies that remain between your sdist package contents and your pex-ified application.
one way to do is:
python setup.py sdist
then run pex
command with -f DIST_DIR
switch
eg. pex $(pip freeze) -o aflaskapp.pex -e 'aflaskapp.app' -f dist -v
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With