How to install python packages without root privileges?

Question

I am using numpy / scipy / pynest to do some research computing on Mac OS X. For performance, we rent a 400-node cluster (with Linux) from our university so that the tasks could be done parallel. The problem is that we are NOT allowed to install any extra packages on the cluster (no sudo or any installation tool), they only provide the raw python itself.

How can I run my scripts on the cluster then? Is there any way to integrate the modules (numpy and scipy also have some compiled binaries I think) so that it could be interpreted and executed without installing packages?

Answer 1

You don't need root privileges to install packages in your home directory. You can do that with a command such as

pip install --user numpy 

or from source

python setup.py install --user 

See https://stackoverflow.com/a/7143496/284795

---

The first alternative is much more convenient, so if the server doesn't have pip or easy_install, you should politely ask the admins to add it, explaining the benefit to them (they won't be bothered anymore by requests for individual packages).

Answer 2

You could create a virtual environment through the virtualenv package.

This creates a folder (say venv) with a new copy of the Python executable and a new site-packages directory, into which you can "install" any number of packages without needing any kind of administrative access at all. Thus, activating the environment through source venv/bin/activate will give Python an environment that's equivalent to having those packages installed.

I know this works for SGE clusters, although how the virtual environment is activated might depend on your cluster's configuration.

You can try installing virtualenv on your cluster within your own site-packages directory using the following steps:

1. Download virtualenv from here, put it on your cluster

2. Install it using setup.py to a specific, local directory to serve as your own site-packages:

   python setup.py build
   python setup.py install --install-base /path/to/local-site-packages
   

3. Add that directory to your PYTHONPATH:

   export PYTHONPATH="/path/to/local-site-packages:${PYTHONPATH}"
   

4. Create a virtualenv:

   virtualenv venv