A simple Hello World setuptools package and installing it with pip

Question

I'm having trouble figuring out how to install my package using setuptools, and I've tried reading the documentation on it and SO posts, but I can't get it to work properly. I'm trying to get a simple helloworld application to work. This is how far I got:

helloworld.py:

print("Hello, World!")

README.txt:

Hello, World! readme

MANIFEST.in:

recursive-include images *.gif

setup.py:

from setuptools import setup, find_packages

setup(
    name='helloworld',
    version='0.1',
    license='BSD',
    author='gyeh',
    author_email='[email protected]',
    url='http://www.hello.com',
    long_description="README.txt",
    packages=find_packages(),
    scripts = ['helloworld.py'],
    package_data={
        "" : ["images/*.gif"]
    },
    data_files=[('images', ['images/hello.gif'])],
    description="Hello World testing setuptools",
)

And I have a blank file called images/hello.gif that I want to include in my package as additional data. The folder structure looks like this:

testsetup/  
|-- helloworld.py  
|-- images/  
|-- --- hello.gif  
|-- MANIFEST.in  
|-- README.txt  
|-- setup.py  

When I run python setup.py sdist, it generates the dist and helloworld.egg-info successfully. When I look at SOURCES.txt under egg-info, it contains the script and the image under the images folder, and the tarball under dist contains them as well.

However, when I try to run pip install --user helloworld-0.1.tar.gz on the tarball, it successfully installs it, but I can't find the program files helloworld.py and images/hello.gif.

When I look under $HOME/.local/lib/python3.3/site-packages/, I see the egg-info folder and all of it's contents installed there. But the $HOME/.local/bin folder doesn't even exist. Are the program files stores elsewhere? What am I doing wrong here? I'm running Arch Linux.

Answer 1

Okay, so after some effort, I finally managed to get a simple "hello world" example working for setuptools. The Python documentation is usually amazing, but I wish the documentation was better on this in particular.

I'm going to write a fairly detailed guide on how I achieved this, and I'll assume no prior background on the reader on this topic. I hope this comes in handy for others...

In order to get this example set up, we'll be creating a package (actually two of them, one for the data files). This is the directory structure we'll end up with:

test-setuptools/
|-- helloworld/
|-- --- hello.py
|-- --- images/
|-- --- --- hello.gif
|-- --- --- __init__.py
|-- --- __init__.py
|-- MANIFEST.in
|-- README.txt
|-- setup.py

Here are the steps:

1. Create the helloworld package.
   1.1 Create the helloworld/ folder as shown in the directory structure above.

   1.2 Add a blank file called __init__.py in the helloworld/ folder.
   If you don't add it, the package won't be recognized (run touch __init__.py to create the file on linux/mac machines). If you want some code to be executed every time the package is imported, include it in the __init__.py file.

   1.3 Create the the hello.py script file to demonstrate the package functionality.

Here is the code for hello.py:

import os

"""
Open additional data files using the absolute path,
otherwise it doesn't always find the file.
"""
# The absolute path of the directoy for this file:
_ROOT = os.path.abspath(os.path.dirname(__file__))

class Hello(object):
    def say_hello(self):
        return "Hello, World!"
    def open_image(self):
        print("Reading image.gif contents:")

        # Get the absolute path of the image's relative path:
        absolute_image_path = os.path.join(_ROOT, 'images/hello.gif')

        with open(absolute_image_path, "r") as f:
            for line in f:
                print(line)

1. 1.4 Create the images/ folder inside the helloworld/ folder.
   Make another blank __init__.py file, because this folder will also be a package.

   1.5 Create the hello.gif file inside the images/ folder.
   This file won't be an actual gif file. Instead, add plain text just to demonstrate that non-script files can be added and read.

I added the following code in hello.gif:

This should be the data inside hello.gif...
...but this is just to demonstrate setuptools,
so it's a dummy gif containing plain text

1. 1.6 Test your package
   Run python from the test-setuptools folder, which will open the python interpreter.
   Type import helloworld.hello to import the hello.py script in the helloworld package. The import should be successful, indicating that you successfully created a package. Make sure that the package in the images/ folder also works, by typing import helloworld.images
   Try instantiating the object that we wrote in hello.py. Type the following commands to make sure everything works as expected:
   hey = helloworld.hello.Hello()
hey.say_hello()
hey.open_image()

2. Create the setup.py file and the remaining files.
   2.1 Create a simple README.txt file. Mine just has the text: Hello, World! Readme inside.

   2.2 Create a MANIFEST.in file with the following contents:
   include helloworld/images/hello.gif
   .
   This is very important because it tells setuptools to include the additional data in the source distribution (which we'll generate in a later step). Without this, you won't be able to install additional, non .py data to your package. See this for additional details and commands.

   2.3 Create the setup.py file (see the code below).
   The most important attributes are packages, include_package_data, and package_data.
   .
   The packages attribute contains a list of the packages you want to include for setuptools. We want to include both the helloworld package and the helloworld.images package that contains our additional data hello.gif.
   You can make setuptools automatically find these by adding the from setuptools import find_packages import and running the imported find_packages() function. Run the interpreter from the test-setuptools folder and test this command to see which packages are found.
   .
   The package_data attribute tells setuptools to include additional data. It's this command, the helloworld.images package, and the MANIFEST.in file which allow you to install additional data.
   The 'helloworld.images' : ['hello.gif'] key/value pair tells setuptools to include hello.gif inside the helloworld.images package if it exists. You can also say '' : ['*.gif'] to include any .gif file in any of the included packages.
   The include_package_data attribute set to True is also necessary for this to work.
   You can include additional metadata for the package like I have (I think an author is necessary). It's a good idea to add classifiers. Additional info can be found here.

Here is the entire setup.py code:

from setuptools import setup

setup(
    name='helloworld',
    version='0.1',
    license='BSD',
    author='gyeh',
    author_email='[email protected]',
    url='http://www.hello.com',
    long_description="README.txt",
    packages=['helloworld', 'helloworld.images'],
    include_package_data=True,
    package_data={'helloworld.images' : ['hello.gif']},
    description="Hello World testing setuptools",
)

.
3. Install and test your package with setuptools.

3.1 Create the source distribution  

Run python setup.py sdist from the test-setuptools/ folder to generate the source distribution.
This will create a dist/ folder containing your package, and a helloworld.egg-info/ folder containing metadata such as SOURCE.txt.
Check SOURCE.txt to see if your the hello.gif image file is included there.
Open the .tar.gz file under the dist/ folder. You should see all of the files described in the directory structure we made earlier, including hello.gif and hello.py.

3.2 Install the distribution  

Install the .tar.gz distribution file by running pip install --user helloworld-0.1.tar.gz from the dist/ folder.
Check that the package was successfully installed by running pip list. The package helloworld should be there.

That's it! now you should be able to test your package under any folder. Open up the interpreter in any folder except test-setuptools, and try importing the package using import helloworld.hello. It should work. Then try the commands to instantiate the object and open the image file using the hey.open_image() command again. It should still work!

You can view exactly which files were installed by pip, and where they are, by uninstalling the package. Mine looked like this:

[gyeh@gyeh package]$ pip uninstall helloworld
Uninstalling helloworld:
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld-0.1-py3.3.egg-info
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/__init__.py
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/__pycache__/__init__.cpython-33.pyc
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/__pycache__/hello.cpython-33.pyc
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/hello.py
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/images/__init__.py
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/images/__pycache__/__init__.cpython-33.pyc
  /home/gyeh/.local/lib/python3.3/site-packages/helloworld/images/hello.gif
Proceed (y/n)? y
  Successfully uninstalled helloworld

As you can see, it successfully installed the additional data file hello.gif, and because we converted the relative path to an absolute path in hello.py, it can read the file just fine.
You can then share this package on PyPI for the rest of the world to use! The instructions on uploading to PyPI are fairly straightforward and can be found here and here.

Once it's online in PyPI, people can search for your package using pip search. Or alternatively, running pip install --user [package-name] will tell pip to search the online PyPI directory for that package name. If it exists, it will install it.

You can run that command for any python package that in PyPI for an easy install so you aren't mucking around with build files.

I hope this saves people a bunch of headaches.