"Too many open files" error when opening and loading images in Pillow

Question

When running the following code:

KEEP=[]
for file in glob.glob("./KEEP/thing*.[tT][iI][fF]"):
    m = pattern.search(file)
    filename=m.group(1)
    keep=Image.open(file)
    keep.load()
    KEEP.append(keep)
    KEEP_NAMES.append(filename)
    keep.close()

over more than a thousand files, I get the error message:

Traceback (most recent call last):
  File "/hom/yannis/texmf/python/remove-harakat.py", line 123, in <module>
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2237, in open
IOError: [Errno 24] Too many open files: './KEEP/thing1118_26.TIF'

I don't understand why this is happening, since I'm load()ing and then close()ing all files, why should they remain open? Is there a solution to this problem, other than reducing the number of files (which is not an option for me)? Some way to close them after their contents have been read in memory?

Answer 1

This may be a bug with the Image.load method - see Pillow issue #1144. I ran into the same too many open files error - see #1237.

My work-around was to load the image into a temp object, make a copy, and then explicitly close the temp. For your code it would look something like this:

KEEP=[]
for file in glob.glob("./KEEP/thing*.[tT][iI][fF]"):
    m = pattern.search(file)
    filename = m.group(1)
    temp = Image.open(file)
    keep = temp.copy()
    KEEP.append(keep)
    KEEP_NAMES.append(filename)
    temp.close()

Answer 2

I also ran into this issue and resolved the issue with a slightly different approach.

This workaround uses copy.deepcopy(), which is based on similar logic of @indirectlylit's solution but avoids creating of temp. See code snippet below

import copy

KEEP=[]
for file in glob.glob("./KEEP/thing*.[tT][iI][fF]"):
    m = pattern.search(file)
    filename = m.group(1)
    keep = copy.deepcopy(Image.open(file))
    KEEP.append(keep)
    KEEP_NAMES.append(filename)