What is f2py used for while building numpy source?

Question

When I list all the Fortran files in NumPy's source tree, I get:

./doc/source/f2py/scalar.f
./doc/source/f2py/string.f
./doc/source/f2py/calculate.f
./doc/source/f2py/moddata.f90
./doc/source/f2py/array.f
./doc/source/f2py/allocarr.f90
./doc/source/f2py/extcallback.f
./doc/source/f2py/common.f
./doc/source/f2py/ftype.f
./doc/source/f2py/fib3.f
./doc/source/f2py/callback.f
./doc/source/f2py/fib1.f
./doc/f2py/f2python9-final/src/examples/exp1.f
./doc/f2py/simple.f
./doc/f2py/multiarray/foo.f
./doc/f2py/hello.f
./doc/f2py/ex1/bar.f
./doc/f2py/ex1/foobar-smart.f90
./doc/f2py/ex1/foo.f
./doc/f2py/ex1/arr.f
./doc/f2py/ex1/foobar.f90
./numpy/f2py/tests/src/mixed/foo_fixed.f90
./numpy/f2py/tests/src/mixed/foo_free.f90
./numpy/f2py/tests/src/mixed/foo.f
./numpy/f2py/tests/src/size/foo.f90
./numpy/f2py/tests/src/kind/foo.f90
./numpy/f2py/tests/src/assumed_shape/precision.f90
./numpy/f2py/tests/src/assumed_shape/foo_use.f90
./numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap
./numpy/f2py/tests/src/assumed_shape/foo_free.f90
./numpy/f2py/tests/src/assumed_shape/foo_mod.f90
./numpy/f2py/src/test/bar.f
./numpy/f2py/src/test/foo.f
./numpy/f2py/src/test/foo90.f90
./numpy/f2py/src/test/wrap.f
./numpy/distutils/tests/f2py_ext/src/fib1.f
./numpy/distutils/tests/f2py_f90_ext/include/body.f90
./numpy/distutils/tests/f2py_f90_ext/src/foo_free.f90

So none other than f2py itself uses Fortran. I looked into the linear algebra modules. For LAPACK, there is a make_lite.py file which extracts only the necessary subroutines from a LAPACK source tree, and converts them into C using f2c. So when exactly during the creation of NumPy was it convenient to create f2py? Am I missing something?

EDIT

It turns out a lot of packages inside SciPy use f2py. Running

$ find . -iname '*.f*'  | cut -d'/' -f3,4 | uniq

gives me the exact directories filled with Fortran files:

linalg/src
fftpack/src
odr/odrpack
special/cdflib
special/amos
special/mach
special/specfun
integrate/quadpack
integrate/odepack
integrate/dop
integrate/linpack_lite
integrate/mach
sparse/linalg
interpolate/fitpack
optimize/minpack2
optimize/minpack
optimize/nnls
optimize/cobyla
optimize/lbfgsb
optimize/slsqp
stats/mvndst.f
stats/futil.f
stats/statlib
_build_utils/src
lib/lapack

Answer 1

OK, after a bit of poking around I think I've confirmed a few of my initial suspicions

First:

So none other than f2py itself uses Fortran.

As I mentioned in the comments, all of the Fortran source files the OP is referring to are in /test/ or /doc/ directories, and I therefore suspect that they are for testing and documenting f2py (and numpy.distutils, which uses f2py). A glance at a few of the source files seems to confirm this impression. f2py itself looks like it's written in Python and C.

I looked into the linear algebra modules. For LAPACK, there is a make_lite.py file which extracts only the necessary subroutines from a LAPACK source tree, and converts them into C using f2c

This seemed strange to me, since I don't actually have f2c installed (or Plex, which is another library that seems to be required by make_lite.py). I decided to stick an extra line in main() to show whether make_lite.py actually gets used during a normal install:

...
def main():

    # let's see if you're actually doing anything
    import subprocess; subprocess.call(['touch', '/tmp/hello_from_make_lite'])
...

Sure enough, after installing numpy in a clean virtualenv there is no hello_from_make_lite file in my /tmp/, indicating that make_lite.main() never executed. Take a look at numpy/linalg/lapack_lite/README:

The numpy/linalg/blas_lite.c, numpy/linalg/dlapack_lite.c, and numpy/linalg/zlapack_lite.c are f2c'd versions of the LAPACK routines required by the LinearAlgebra module, and wrapped by the lapack_lite module. The scripts in this directory can be used to create these files automatically from a directory of LAPACK source files.

So numpy is already distributed with these f2c'd C source files - there's no need to use make_lite.py unless you're a dev wanting to update these functions from a new version of the LAPACK library.

So when exactly during the creation of NumPy was it convenient to create f2py?

As far as I can tell, f2py doesn't get used at all during a normal numpy install. Again, I stuck an extra line in f2py2e.main():

...
def main():
    import subprocess; subprocess.call(['touch', '/tmp/hello_from_f2py2e'])
...

And again, /tmp/hello_from_f2py2e doesn't exist after a normal install of numpy.

So what is f2py actually used for? Check out the scipy source tree, and from its root call

$ find . -iname *.f*

You'll see loads and loads of important-looking Fortran files, including fftpack, odepack, arpack, fitpack, etc. I suspect that f2py is mainly required to wrap the Fortran extensions for scipy rather than numpy.

I might be wrong, though - perhaps one of the numpy or scipy devs will set me straight.

Update

Actually, I think f2py is not actually required during a normal installation of scipy either! If you take a look in the source directory for one of the Fortran modules, e.g. fftpack, you'll see that it already contains .pyf files, which would normally be automatically generated by f2py and define the interfaces for the Fortran functions (see here).

I think the deal is that f2py was used to initially generate the .pyf wrappers for the Fortran functions, but these .pyf files are distributed along with the rest of the source tree so that it's unnecessary to run f2py again during the normal build process.