How can you bundle all your python code into a single zip file?

Question

It would be convenient when distributing applications to combine all of the eggs into a single zip file so that all you need to distribute is a single zip file and an executable (some custom binary that simply starts, loads the zip file's main function and kicks python off or similar).

I've seen some talk of doing this online, but no examples of how to actually do it.

I'm aware that you can (if it is zip safe) convert eggs into zip files.

What I'm not sure about is:

Can you somehow combine all your eggs into a single zip file? If so, how?

How would you load and run code from a specific egg?

How would you ensure that the code in that egg could access all the dependencies (ie. other eggs in the zip file)?

People ask this sort of stuff a lot and get answers like; use py2exe. Yes, I get it, that's one solution. It's not the question I'm asking here though...

Answer 1

You can automate most of the work with regular python tools. Let's start with clean virtualenv.

[zart@feena ~]$ mkdir ziplib-demo [zart@feena ~]$ cd ziplib-demo [zart@feena ziplib-demo]$ virtualenv . New python executable in ./bin/python Installing setuptools.............done. Installing pip...............done. 

Now let's install set of packages that will go into zipped library. The trick is to force installing them into specific directory.

(Note: don't use --egg option either on command-line or in pip.conf/pip.ini because it will break file layout making it non-importable in zip)

[zart@feena ziplib-demo]$ bin/pip install --install-option --install-lib=$PWD/unpacked waitress Downloading/unpacking waitress   Downloading waitress-0.8.5.tar.gz (112kB): 112kB downloaded   Running setup.py egg_info for package waitress  Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from waitress) Installing collected packages: waitress   Running setup.py install for waitress      Installing waitress-serve script to /home/zart/ziplib-demo/bin Successfully installed waitress Cleaning up... 

Update: pip now has -t <path> switch, that does the same thing as --install-option --install-lib=.

Now let's pack all of them into one zip

[zart@feena ziplib-demo]$ cd unpacked [zart@feena unpacked]$ ls waitress  waitress-0.8.5-py2.7.egg-info [zart@feena unpacked]$ zip -r9 ../library.zip *   adding: waitress/ (stored 0%)   adding: waitress/receiver.py (deflated 71%)   adding: waitress/server.pyc (deflated 64%)   adding: waitress/utilities.py (deflated 62%)   adding: waitress/trigger.pyc (deflated 63%)   adding: waitress/trigger.py (deflated 61%)   adding: waitress/receiver.pyc (deflated 60%)   adding: waitress/adjustments.pyc (deflated 51%)   adding: waitress/compat.pyc (deflated 56%)   adding: waitress/adjustments.py (deflated 60%)   adding: waitress/server.py (deflated 68%)   adding: waitress/channel.py (deflated 72%)   adding: waitress/task.pyc (deflated 57%)   adding: waitress/tests/ (stored 0%)   adding: waitress/tests/test_regression.py (deflated 63%)   adding: waitress/tests/test_functional.py (deflated 88%)   adding: waitress/tests/test_parser.pyc (deflated 76%)   adding: waitress/tests/test_trigger.pyc (deflated 73%)   adding: waitress/tests/test_init.py (deflated 72%)   adding: waitress/tests/test_utilities.pyc (deflated 78%)   adding: waitress/tests/test_buffers.pyc (deflated 79%)   adding: waitress/tests/test_trigger.py (deflated 82%)   adding: waitress/tests/test_buffers.py (deflated 86%)   adding: waitress/tests/test_runner.py (deflated 75%)   adding: waitress/tests/test_init.pyc (deflated 69%)   adding: waitress/tests/__init__.pyc (deflated 21%)   adding: waitress/tests/support.pyc (deflated 48%)   adding: waitress/tests/test_utilities.py (deflated 73%)   adding: waitress/tests/test_channel.py (deflated 87%)   adding: waitress/tests/test_task.py (deflated 87%)   adding: waitress/tests/test_functional.pyc (deflated 82%)   adding: waitress/tests/__init__.py (deflated 5%)   adding: waitress/tests/test_compat.pyc (deflated 53%)   adding: waitress/tests/test_receiver.pyc (deflated 79%)   adding: waitress/tests/test_adjustments.py (deflated 78%)   adding: waitress/tests/test_adjustments.pyc (deflated 74%)   adding: waitress/tests/test_server.pyc (deflated 73%)   adding: waitress/tests/fixtureapps/ (stored 0%)   adding: waitress/tests/fixtureapps/filewrapper.pyc (deflated 59%)   adding: waitress/tests/fixtureapps/getline.py (deflated 37%)   adding: waitress/tests/fixtureapps/nocl.py (deflated 47%)   adding: waitress/tests/fixtureapps/sleepy.pyc (deflated 44%)   adding: waitress/tests/fixtureapps/echo.py (deflated 40%)   adding: waitress/tests/fixtureapps/error.py (deflated 52%)   adding: waitress/tests/fixtureapps/nocl.pyc (deflated 48%)   adding: waitress/tests/fixtureapps/getline.pyc (deflated 32%)   adding: waitress/tests/fixtureapps/writecb.pyc (deflated 42%)   adding: waitress/tests/fixtureapps/toolarge.py (deflated 37%)   adding: waitress/tests/fixtureapps/__init__.pyc (deflated 20%)   adding: waitress/tests/fixtureapps/writecb.py (deflated 50%)   adding: waitress/tests/fixtureapps/badcl.pyc (deflated 44%)   adding: waitress/tests/fixtureapps/runner.pyc (deflated 58%)   adding: waitress/tests/fixtureapps/__init__.py (stored 0%)   adding: waitress/tests/fixtureapps/filewrapper.py (deflated 74%)   adding: waitress/tests/fixtureapps/runner.py (deflated 41%)   adding: waitress/tests/fixtureapps/echo.pyc (deflated 42%)   adding: waitress/tests/fixtureapps/groundhog1.jpg (deflated 24%)   adding: waitress/tests/fixtureapps/error.pyc (deflated 48%)   adding: waitress/tests/fixtureapps/sleepy.py (deflated 42%)   adding: waitress/tests/fixtureapps/toolarge.pyc (deflated 43%)   adding: waitress/tests/fixtureapps/badcl.py (deflated 45%)   adding: waitress/tests/support.py (deflated 52%)   adding: waitress/tests/test_task.pyc (deflated 78%)   adding: waitress/tests/test_channel.pyc (deflated 78%)   adding: waitress/tests/test_regression.pyc (deflated 68%)   adding: waitress/tests/test_parser.py (deflated 80%)   adding: waitress/tests/test_server.py (deflated 78%)   adding: waitress/tests/test_receiver.py (deflated 87%)   adding: waitress/tests/test_compat.py (deflated 51%)   adding: waitress/tests/test_runner.pyc (deflated 72%)   adding: waitress/__init__.pyc (deflated 50%)   adding: waitress/channel.pyc (deflated 58%)   adding: waitress/runner.pyc (deflated 54%)   adding: waitress/buffers.py (deflated 74%)   adding: waitress/__init__.py (deflated 61%)   adding: waitress/runner.py (deflated 58%)   adding: waitress/parser.py (deflated 69%)   adding: waitress/compat.py (deflated 69%)   adding: waitress/buffers.pyc (deflated 69%)   adding: waitress/utilities.pyc (deflated 60%)   adding: waitress/parser.pyc (deflated 53%)   adding: waitress/task.py (deflated 72%)   adding: waitress-0.8.5-py2.7.egg-info/ (stored 0%)   adding: waitress-0.8.5-py2.7.egg-info/dependency_links.txt (stored 0%)   adding: waitress-0.8.5-py2.7.egg-info/installed-files.txt (deflated 83%)   adding: waitress-0.8.5-py2.7.egg-info/top_level.txt (stored 0%)   adding: waitress-0.8.5-py2.7.egg-info/PKG-INFO (deflated 65%)   adding: waitress-0.8.5-py2.7.egg-info/not-zip-safe (stored 0%)   adding: waitress-0.8.5-py2.7.egg-info/SOURCES.txt (deflated 71%)   adding: waitress-0.8.5-py2.7.egg-info/entry_points.txt (deflated 33%)   adding: waitress-0.8.5-py2.7.egg-info/requires.txt (deflated 5%) [zart@feena unpacked]$ cd .. 

Note that those files should be at top of zip, you can't just zip -r9 library.zip unpacked

Checking the result:

[zart@feena ziplib-demo]$ PYTHONPATH=library.zip python Python 2.7.1 (r271:86832, Apr 12 2011, 16:15:16) [GCC 4.6.0 20110331 (Red Hat 4.6.0-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import waitress >>> waitress <module 'waitress' from '/home/zart/ziplib-demo/library.zip/waitress/__init__.pyc'> >>> >>> from wsgiref.simple_server import demo_app >>> waitress.serve(demo_app) serving on http://0.0.0.0:8080 ^C>>> 

Update: since python 3.5 there is also zipapp module which can help with bundling the whole package into .pyz file. For more complex needs pyinstaller, py2exe or py2app might better fit the bill.

Answer 2

You could use the zipapp module from the standard library to create executable Python zip archives. It is available from Python 3.5 onwards.

One way to create a bundle is to add a top-level file named __main__.py, which will be the script that Python runs when the zip executable archive is executed.

Suppose your directory structure is now like this:

└── myapp     ├── __main__.py     ├── myprog1.py     └── myprog2.py 

If your code has external dependencies (e.g. listed in a file named requirements.txt), install them into the directory using:

pip3 install -r requirements.txt --target myapp/ 

note 1: This will fill the myapp/ directory with the external dependencies.

note 2: Debian/Ubuntu users may need to use the --system option for pip3, because the Debian/Ubuntu version of pip seems to use --user by default.

Then, create the zip executable archive using:

python3 -m zipapp myapp/ 

This will create a zip executable archive named myapp.pyz, which you can execute by running:

python3 myapp.pyz 

When the zip executable archive is executed, it is __main__.py that is run.

If, in addition to Python scripts, you need to include other data files (e.g. text files, PNG images, etc.) used by the Python scripts, see: python: can executable zip files include data files?