Enthought Python, Sage, or others (in Unix clusters)

Question

I have access to a cluster of Unix machines, but they don't have the software I need (numpy, scipy, matplotlib, etc), so I have to install them by myself (I don't have root permissions, either, so commands like apt-get or yast don't work).

In the worst case, I will have to compile them all from source. Is there any better way to proceed? I've heard something about Enthought Python and Sage, but am not sure what is the best way to go.

Any suggestions?

Answer 1

EPD (Enthought Python Distribution) is great, but even for academics, you can only get the 32-bit version free of charge. If you intend to do anything ram-intensive, it's not really an option.

Edit: This has since changed, and the 64-bit version is freely available for academic/educational use.

On the other hand, the Intel MLK library does make a difference, and it has lots of nifty (e.g. the latest version of mayavi) things built that can otherwise be a real pain to build from source. Also, as others have said, you can just untar it into your home folder and run it. You shouldn't need root access.

EPD is an absolutely great option if you don't ever need to use more than 2GB of ram, but you will have to pay to get 64-bit builds.

Python(x,y) is great if you're on windows, but otherwise, good luck finding the linux pre-built binaries. They more-or-less no longer exist... The ubuntu repository seems to be permanently down, and I don't know of anywhere to get a pre-compiled tarball for it anymore. This may all change in the near future, though... Hopefully it does, because it would be a great option for you!

Honestly, if you just need numpy, scipy, and matplotlib, they're relatively easy to build from source (especially so if you can get away without scipy), and you can always just build your own python interpreter and then use easy_install to avoid having to build them from source. This, of course, assumes that a basic build environment (gcc, etc) are already installed on the machine you're using... That's what I've done when I was in your situation, anyway...

If you go that route, it's best to download the python source code and build your own python interpreter that you'll use for everything. Then install setuptools and easy_install the rest. (Alternately, you can download the source code for numpy, etc and build and install them in for the python interpreter you just built.)

This shows a basic idea how to build the basics (python, numpy, scipy, matplotlib, ipython) under a directory called "pythondist" in the current working directory.

#! /bin/sh

builddir=$(pwd)/pythondist
mkdir -p $builddir/source
cd $builddir/source

wget 'http://python.org/ftp/python/2.6.5/Python-2.6.5.tgz'
wget 'http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e'
tar -xvzf Python-2.6.5.tgz

# Build python
cd $builddir/source/Python-2.6.5/

# The --prefix argument is the key!
./configure --prefix=$builddir

# Be sure to speed things up with the -j option if you're 
# on a multicore machine (e.g. make -j 4 build for a quadcore)
make build 
make install

# Now install setuptools
cd $builddir/source
tar -xvzf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11/

# The next key is to call this with the python you just built!
$builddir/bin/python setup.py build
$builddir/bin/python setup.py install

# Now just install numpy, scipy, ipython, matplotlib, etc through easy_install
$builddir/bin/easy_install numpy
$builddir/bin/easy_install scipy
$builddir/bin/easy_install matplotlib
$builddir/bin/easy_install ipython

EDIT: Minor typos in script. If numpy or scipy doesn't install properly from the egg, see the install notes.

This script is mainly intended to demonstrate building an independent python in your home directory, and assumes the system you're building on has the proper dependencies already installed, but it at least points you in the right direction.

If numpy or scipy don't build properly using easy_install, download the source tarballs and try building building them from there using different arguments. (Numpy/Scipy's setup.py autodetecting the wrong fortran compiler is common problem, in my experience) E.g.

cd $builddir/source
wget http://sourceforge.net/projects/numpy/files/NumPy/1.4.1/numpy-1.4.1.tar.gz/download
tar -xvzf numpy-1.4.1.tar.gz
cd numpy-1.4.1/
# If you don't specify an action (e.g. "build") this will enter an interactive
# mode to help diagnose problems... See the INSTALL.txt file, too!
$builddir/bin/python setup.py 

For example, on my OpenSUSE 11.2 system, I need to specify "--fcompiler=gnu95" when building numpy and scipy, as I have both g77 and gfortran installed. Otherwise things won't build correctly.

However, on an older RHEL 3 system, it builds perfectly as-is from easy_install. YMMV, of course. Good luck!