Pants includes OS X specific Python wheels

Question

TLDR: Pants fetches OS X specific wheels bc I'm developing on Mac. How can I avoid this, or specify that I will deploy to Ubuntu?

Full story:

Trying to package a Python application with Pants. Going great so far, but ran into a problem which I've been stuck at for a while. I'm developing on a macbook but deploying to EC2 Ubuntu.

Here's what I've done so far:

1. Created virtualenv.
2. Added BUILD files to applications, with the suggested 3rd party pattern for third party packages.
3. Ran ./pants run.py backend:admin_server which runs fine and generated dist/admin_server.pex
4. Scp that .pex onto a fresh EC2 Ubuntu box.

However when I run the application there, I get:

Failed to execute PEX file, missing compatible dependencies for:
    mysql-python
    pycrypto

The problem seems to be that Pants takes OS X specific wheels for these 2:

pex: - MySQL_python-1.2.5-cp27-none-macosx_10_11_intel.whl pex: - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl

How can I avoid that, or specify which OS they should run on?

Here's the full output:

ubuntu@ip-***:~$ export PEX_VERBOSE=1
ubuntu@ip-***:~$ python admin_server.pex
pex: Found site-library: /usr/local/lib/python2.7/dist-packages
pex: Found site-library: /usr/lib/python2.7/dist-packages
pex: Tainted path element: /usr/local/lib/python2.7/dist-packages
pex: Tainted path element: /usr/lib/python2.7/dist-packages
pex: Scrubbing from site-packages: /usr/local/lib/python2.7/dist-packages
pex: Scrubbing from site-packages: /usr/lib/python2.7/dist-packages
pex: Scrubbing from user site: /home/ubuntu/.local/lib/python2.7/site-packages
pex: Failed to resolve a requirement: MySQL-python==1.2.5
pex: Failed to resolve a requirement: pycrypto==2.6.1
pex: Unresolved requirements:
pex:   - mysql-python
pex:   - pycrypto
pex: Distributions contained within this pex:
pex:   - six-1.10.0-py2.py3-none-any.whl
pex:   - protobuf-2.6.1-py2.7.egg
pex:   - setuptools-19.5-py2.py3-none-any.whl
pex:   - MySQL_python-1.2.5-cp27-none-macosx_10_11_intel.whl
pex:   - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl
pex:   - futures-3.0.4-py2-none-any.whl
pex:   - webapp2-2.5.2-py2-none-any.whl
pex:   - requests-2.9.0-py2.py3-none-any.whl
pex:   - jmespath-0.9.0-py2.py3-none-any.whl
pex:   - beautifulsoup4-4.4.1-py2-none-any.whl
pex:   - python_dateutil-2.4.2-py2.py3-none-any.whl
pex:   - boto3-1.2.3-py2.py3-none-any.whl
pex:   - WebOb-1.5.1-py2.py3-none-any.whl
pex:   - cssutils-1.0.1-py2-none-any.whl
pex:   - webapp2_static-0.1-py2-none-any.whl
pex:   - Paste-2.0.2-py2-none-any.whl
pex:   - docutils-0.12-py2-none-any.whl
pex:   - botocore-1.3.22-py2.py3-none-any.whl
pex:   - protobuf_to_dict-0.1.0-py2-none-any.whl
Failed to execute PEX file, missing compatible dependencies for:
mysql-python
pycrypto

PS: to make sure I didn't include my versions of the python libraries, I pip uninstalled both PyCrypto and MySQL-Python.

Answer 1

One of the nice things about distributing your project as a PEX file is that you can prepare it to run on multiple platforms. For example, one PEX can run on both Linux and Mac platforms. For many projects, there is nothing special to do other than build a PEX. But when your project has dependencies on platform specific binary code, you will need to perform some extra steps.

One example of a library that contains platform specific code is the psutil library. It contains C code that it compiled into a shared library when the module is installed. To create a PEX file that contains such dependencies, you must first provide a pre-built version of that library for all platforms other than the one where you are running pants.

The easiest way to pre-build libraries is to use the pip tool to build wheels.

This recipe assumes the following:

• You want to build a multi-platform pex to run on both Linux and mac You are going to pre-build the libraries in the Linux environment, then build the PEX on the mac environment.
• Your project directory lives under ~/src/cookbook

Let’s take a simple program that references a library and create a pex from it.

#  src/python/ps_example/main.py
import psutil

for proc in psutil.process_iter():
    try:
        pinfo = proc.as_dict(attrs=['pid', 'name'])
    except psutil.NoSuchProcess:
        pass
    else:
        print(pinfo)

With Pants, you can define an executable by defining a python_binary target in a BUILD file:

# src/python/ps_example/BUILD
python_binary(name='ps_example',
  source = 'main.py',
  dependencies = [
    ':psutil',  # defined in requirements.txt
  ],
)

# Defines targets from specifications in requirements.txt
python_requirements()

In the same directory, list the python libraries in a requirements.txt file:

# src/python/ps_example/requirements.txt 
psutil==3.1.1

Now, to to make the multi-platform pex, you'll need access to a Linux box to create the linux version of psutil wheel. Copy the requirements.txt file to the linux machine, then, execute the pip tool:

linux $ mkdir ~/src/cookbook/wheelhouse
linux $ pip wheel -r src/python/multi-platform/requirements.txt  \
    --wheel-dir=~/src/cookbook/wheelhouse

This will create a platform specific wheel file.

linux $ ls ~/src/cookbook/wheelhouse/
psutil-3.1.1-cp27-none-linux_x86_64.whl

Now, you will need to copy the platform specific wheel over to the machine where you want to build your multi-platform pex (in this case, your mac laptop). If you use this recipe on a regular basis, you will probably want to configure a Python Respository to store your pre-built libraries.

We’ll use the same BUILD file setup as in above, but modify python_binary to specify the platforms= parameter.

# src/python/ps_example/BUILD
python_binary(name='ps_example',
  source = 'main.py',
  dependencies = [
    ':psutil',  # defined in requirements.txt
  ],
  platforms=[
    'linux-x86_64',
    'macosx-10.7-x86_64',
  ],
)

# Defines targets from specifications in requirements.txt
python_requirements()

You will also need to tell pants where to find the pre-built python packages. Edit pants.ini and add:

[python-repos]
repos: [
    "%(buildroot)s/wheelhouse/"
  ]

Now, copy the file psutil-3.1.1-cp27-none-linux_x86_64.whl over to the mac workstation and place it in a directory named wheelhouse/ under the root of your repo.

Once this is done you can now build the multi-platform pex with

mac $ ./pants binary src/python/py_example

You can verify that libraries for both mac and Linux are included in the pex by unzipping it:

mac $ unzip -l dist/ps_example.pex | grep psutil
    17290  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/DESCRIPTION.rst
    19671  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/METADATA
     1340  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/RECORD
      103  12-21-15 22:09  
...   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/DESCRIPTION.rst
    19671  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/METADATA
     1338  12-21-15 22:09   .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/RECORD
      109  12-21-15 22:09   
...