How can I find out why using PIL functions is resulting in a memory error?

Question

I am using PIL (python image library) to do some image manipulation, specifically I am stitching images together.

My code seems to work fine for some images in small quantities, but sometime I and getting a MemoryError.

The part that is particularly strange to me is that I am not doing manipulations on bit images, its all work with <10 images under 10kb.

I am making a lot of calls to Image.resize, but I am surprised that there are significant issues from that.

Here is the stack track:

Traceback (most recent call last):                                                                                                                                               
  File "test.py", line 15, in <module>                                                                                                                                           
    pprint(scale_matrix_down((90,90), [inpt]))                                                                                                                                   
  File "/Users/jeremykarmel/Desktop/Python/merger.py", line 105, in scale_matrix_down                                                                                            
    return [shrinkRow(row, row_width_overflow(row)) for row in matrix]                                                                                                           
  File "/Users/jeremykarmel/Desktop/Python/merger.py", line 103, in shrinkRow                                                                                                    
    rest         = [shrinkIm(im, pixels_per_im) for im in row[remaining_pixels:]]                                                                                                
  File "/Users/jeremykarmel/Desktop/Python/merger.py", line 110, in shrinkIm                                                                                                     
    return im.resize((im.size[0] - num_pix, im.size[1] - num_pix))                                                                                                               
  File "/Library/Python/2.7/site-packages/PIL/Image.py", line 1302, in resize                                                                                                    
    im = self.im.resize(size, resample)                                                                                                                                          
MemoryError

Keep in mind the images are all less than 90x90 pixels.

I am very much at a dead end and really not sure how to proceed. What can I do to release the memory? Should I be calling the del operator or is there something simpler I can do? Thanks in advance for your help!

Answer 1

To explain what actually happens:

1. You try to resize an image to a negative width
2. PIL tries to allocate a negatively sized image to fit the resized image
3. PIL calls malloc with a negative amount of memory to hold to negatively sized image
4. malloc takes a size_t which is unsigned. As a result, the negative memory request gets interpreted as a very large number rather then a negative number.
5. malloc cannot allocate something large, so it produces an error
6. To PIL this looks exactly like an out of memory error so it reports it as such

So the invalid inputs aren't causing leaks or other bad behavior. They result in an bad malloc request and give up. PIL could check for negative sizes, and thus produce a better error message. Perhaps they figure its not worth it because it already produces an error message if you do that.

As it happens, actually running out memory is hard because the OS will try pretty hard to keep a process going such as by using virtual memory. Your system will bog down before it gets to a point of running out memory. So I've found that most out-of-memory errors are caused by requests for negative amounts of memory. As far as I recall, the only real out-of-memory errors I've gotten have been in Java probably due to its use of a virtual machine.

Answer 2

It turns out this is not actually a memory error. As Winston Ewert pointed out I was in fact feeding negative parameters into an images resize method.

Even though the python documentation says that memory errors are for problems with memory, this error gets thrown when you give negative params to resize. My suspicion is that because PIL heavily leverages C libraries, those can lead to leaks with invalid input and since the library does not do bounds checking, that the error simply bubbles up.