OSError 24 (Too many open files) when reading bunch of FITS with astropy.io

Question

I’m trying to load into memory a few 2 000 FITS using astropy.io.fits:

def readfits(filename):
    with fits.open(filename) as ft:
        # the fits contain a single HDU
        data = ft[0].data
    return data

data_sci = []
for i in range(2000):
    data_sci.append(readfits("filename_{}.fits".format(i)))

However, when reaching the 1015th file, OSError: [Errno 24] Too many open files is raised.

I have the same issue with:

def readfits(filename):
    ft = fits.open(filename) as ft:
    data = ft[0].data
    ft.close()
    return data

I suspect that astropy.io.fits does not properly close the file. Is there a way I can force the files to be closed?

Answer 1

Your readfits function actually leaves the file handle open in order to keep access to the data, because by default it creates a mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Incidentally, if you just want a function that reads the data out of the first HDU this is already built in: http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata

It's not necessary to reinvent the wheel.

Answer 2

After taking a look at the astropy documentation i found this: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files