Python Pillow opening 1-bit depth files with 8 bits

Question

I've got about 7GB of single-bit depth images (black and white BMP files), around 59 gigapixels total, which I need to paste together into a single square image. My system has 64GB RAM, so I figured this shouldn't be a problem to load all images at once, then create a new output image (should be the same size/resource-load), and paste the sub-images into it. Finally saving it to disk.

Well, watching top, I see after my images are loaded, the program is consuming 56GB RAM (i.e. 7GB * 8). During the pasting operation, the program gets killed.

Can I somehow change the Pillow open call to open in the native depth? Is there another image processing library someone might recommend instead (other languages are acceptable if that's the only obvious route...).

Here's the code:

import os
import math
import numpy as np
from rectpack import newPacker, PackingMode
import rectpack.packer
from PIL import Image, PILLOW_VERSION
dirname = 'output_12_15'
files = sorted(os.listdir(dirname))
images = []
num_pixels = 0
for i, file in enumerate(files[1:]):
    filepath = os.path.join(dirname, file)
    img = Image.open(filepath)
    images.append(img.copy())
    img.close()
    width, height = images[-1].size
    num_pixels += width*height
    if not i%1000:
        print(i)
print('number of total pixels: {}'.format(num_pixels))
edge_len = math.ceil(num_pixels**(1/2))
theorhetical_num_images_wide = math.ceil(edge_len/1366.)
target_width = (theorhetical_num_images_wide+1)*1366
target_height = target_width

packer = newPacker(mode=PackingMode.Offline, sort_algo=rectpack.packer.SORT_NONE, rotation=False)
# Add the rectangles to packing queue
for img in images:
    packer.add_rect(*img.size)
packer.add_bin(target_width, target_height)
print('starting to pack')
packer.pack()
print('finished packing, starting to paste image together')

if len(packer[0])!=len(images):
    print('packer failed, bin was too small!')
    import pdb;pdb.set_trace()
else:
else:
    max_y = 0
    max_x = 0
    blank = Image.new("1", (target_width,target_height))
    for i, rect in enumerate(packer[0]):
        x = rect.x
        y = rect.y
        w = rect.width
        h = rect.height
        if y+h>max_y:
            max_y = y+h
        if x+w>max_x:
            max_x = x+w
        blank.paste(images[i], (x,y))
    print('max_x {} max_y {} target_h {}'.format(max_x, max_y, target_height))
    blank = blank.convert('1', dither=Image.NONE)
    blank.save("59MP.bmp", "BMP")

UPDATE 1: I moved the code to another system with 256GB RAM, and found a bug after things got a little further (into the last FOR loop). But then it crashes again with this traceback:

0
1000
2000
3000
4000
5000
6000
number of total pixels: 59926682272
starting to pack
finished packing, starting to paste image together
max_x 247246 max_y 247243 target_h 247246
Traceback (most recent call last):
  File "stitch_bmps.py", line 75, in <module>
    blank.save("59MP.bmp", "BMP")
  File "/home/nmz787/.local/lib/python3.7/site-packages/PIL/Image.py", line 2084, in save
    save_handler(self, fp, filename)
  File "/home/nmz787/.local/lib/python3.7/site-packages/PIL/BmpImagePlugin.py", line 332, in _save
    + o32(offset)  # reserved
  File "/home/nmz787/.local/lib/python3.7/site-packages/PIL/_binary.py", line 91, in o32le
    return pack("<I", i)
struct.error: 'I' format requires 0 <= number <= 4294967295

UPDATE 2: added a pdb.set_trace() call in the BmpImagePlugin file just before the error:

(Pdb) image
7641879368
(Pdb) offset+image
7641879430
(Pdb) (offset+image)<4294967295
False
(Pdb) (image)<4294967295
False

Sooo, I guess this is saying that BMP format, or the implementation PIL has for BMP, doesn't support such large images?

Answer 1

According to https://pillow.readthedocs.io/en/5.1.x/handbook/concepts.html, there is only an image mode

1 (1-bit pixels, black and white, stored with one pixel per byte)

but not one with 8 pixels per byte.

So it seems this is not possible with Pillow.

Answer 2

single-bit depth images(...)Sooo, I guess this is saying that BMP format, or the implementation PIL has for BMP, doesn't support such large images?

Then I suggest trying PBM file format, which is textual description of black-and-white image, let file.pbm content be

P1
7 7
0000000
0111110
0100010
0100010
0100010
0111110
0000000

observe that it consist of header (P1) then dimensions (7 7) and then pixels marked with digits 0 and 1. Simplicity of this format allows you to write just like text file or you might use netpbmfile to read such file as numpy array and save numpy array to such file. PBM file format is supported by ImageMagick, therefore you might convert it to another format which is required downstream.