Java - How to write a very large (20,000x20,000 px or larger) tif image

Question

I am working with extremely large tif images that I am composing into a large single image. I have a library that was created by a colleague of mine that generates the image pyramid and provides a very handy tool for visualizing the image pyramid. This visualizer is great for taking a peak at the large image and visually identifying points of interest, but the customers are more interested in image analysis on these large images.

Thus, it is necessary to export the very large image into a single file. I find this to be troublesome considering these images can be anywhere from 800 MB to multiple GBs in size. And just the task of loading this single image to memory is challenging, particularly when image analysis is being done.

I was wondering, if it were possible in java to write this large tiff image in a block or line-by-line fashion. Currently my application is running out of memory on small (8 GB RAM) machines.

The current method for composing these images is:

1. Store the pixel values into a BufferedImage using a WritableRaster

   short[][]pixels = ...
   BufferedImage image = new BufferedImage(width, height, type);
   WritableRaster = image.getRaster();
   for (int row = 0; row < height; row++)
   {
       for (int col = 0; col < width; col++)
       {
          raster.setSample(col, row, 0, pixel[row][col]);
       }
   }   
   

2. And then write the buffered image to disk. For this part I am using ImageJ to write the image as a tif. If there are better ways that support 16-bit grayscale tif images, then I will be happy to take a look

   // BufferedImage image; from above
   ...
   ImagePlus img = new ImagePlus();
   img.setImage(image);
   FileSaver fs = new FileSaver(img);
   fs.saveAsTiff(file.getAbsolutePath());  
   

The problem with this method is it has too large of a memory footprint for an 8GB of RAM machine.

Ideally what i'd like is to just have a single short[][]pixels. This is mainly because I need to compute an average blending function, so some memory footprint will be there. Also in the future I will be adding a linear blend. The short[][]pixels should only take up ~765 MB of RAM for 20k x 20k pixel data, which I think is currently unavoidable, so for larger images for example 100k x 100k pixels, I would hope that the biologists would not want to export this image as it would take up 18GB of RAM.

Later I will modify the code to support exporting the extremely large images like the 100k x 100k. For now I am okay with assuming one piece of memory to store the initial pixel values.

So, what is a good method for writing portions of a tif image to disk so I can support writing out of core images such as the 100k x 100k images.

I did see this post: Write tiled output of TIFF, using ImageIO in Java

But it just discusses the TIFF 6.0 spec. But I will look into ImageOutputStreams. Tif is a beast though so I might just bite the bullet and encourage the biologists to only export the regions of interest.

EDIT: Found a viable solution:

WRITER: https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/out/TiffWriter.java

and

READER: https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/in/TiffReader.java

Main group page: https://github.com/openmicroscopy/bioformats

Answer 1

With the SCIFIO project, we are generalizing the Bio-Formats image I/O framework to target scientific imaging in general, beyond just microscopy and the life sciences. The SCIFIO API is in beta now, and includes TIFF which can of course read and write in tiles. Feedback on API and bugs on the SCIFIO mailing list is always welcome!