Bitmap Performance-Optimization Patterns

Question

I found several patterns for optimizing Bitmaps handling in WPF. I do not, however, understand when to use each patterns. As I think this is a common problem, I summarized what I understand and what I guess and ask for your help. If you can add patterns, explain how they differ, explain if they use the CPU or the GPU, and teach when to use each and how to combine them, it’d be a tremendous help!

Context – the Images "Grid" Scenario:

My application has to display many bitmap images. The images are displayed on the screen in a rows-and-columns grid-like organization (not necessarily the Grid or UniformGrid classes, think Window Media Player’s Album view). Images might move between different grid cells. Some images at arbitrary cells may be replaced by others. Images should be clickable, should provide a context menu, should be selectable, drag-able etc. In other words, “combine the little buggers into one big bitmap” is not applicable, at least not naively.

Pattern 0: The Hack

Do combine the little buggers into a bitmap (how? drawing context?), and use this as the background. Overlay this with images with empty content that will handle the hits, context menus, events etc.

The advantage is that we're only speaking about two bitmaps here: The currently displayed one and the one that should replace it. This should be really fast. However, my years of experience raise the red flag of danger. Your comments?

Pattern 1: Reduce Image Size

This is a no-brainer when you know in advance the image size to resize to, and when you’re prepared to lose details (color) for performance:

1. Reduce the bitmap size using BitmapImage.DecodePixelWidth
2. Reduce the color information using FormatConvertedBitmap.DestinationFormat
3. Set the control’s scaling behavior setting Image.Stretch to Stretch.None
4. Set the SetBitmapScalingMode for the image to LowQuality.
5. Freeze the bugger

See code here.

Pattern 2: Background pre-fetch

This pattern is applicable when you think you can take advantage of the user gazing at the images on the screen, and prepare ahead the next images to display. The cons for your project, in addition to the memory overhead, is that it has to support the .Net Framework 4 target and not just the client profile, so it might incur an installation on the client’s. You yourself will have to suffer the async programming pain.

In this pattern you create exactly the required number of Image controls. When bitmaps need to be added, moved or deleted you only modify the Image controls' BitmapSource(s). A BackgroundWorker task is responsible for pre-fetching the BitmapSource(s) (possibly using the “Reduce Image Size” pattern above) and inserting them into MemoryCache.

For this to work you have to set the BitmapImage’s CacheOption to OnLoad, so that the work is off-loaded to the background worker.

Pattern 3: Drawing Context

This was suggested by Sheldon Ziao from Microsoft Support on the MSDN WPF forum here. See page 494, Chapter 15 “2D Graphics” in Adam Nathan’s WPF 4 Unleashed for a description of DrawingContext. I can’t say I understand it. According to the answer here, I would assume this would improve handling of Geometry drawings, not bitmaps. Next, I don’t think this will support the focus and events requirements for the images (my bad for not explaining the requirements better at the forum) Moreover, I’m worried by the book’s summary sentence: “Note that the use of DrawingContext doesn’t change the fact that you’re operating within a retained-mode system. The specified drawing doesn’t happen immediately; the commands are persisted by WPF until they are needed.” This means that once our even handler re we can’t take advantage of parallelism as in “Background pre-fetch”.

Pattern 4: Writeable Bitmaps

The MSDN documentation here describes it as a dual buffer system: Your UI thread updates the buffer; the WPF’s render thread moves this to video memory.

The intended usage (see here) is for bitmaps that change a lot such in a video movie like display. I’m not sure, but possible this could be hacked and combined with the Background Pre-fetch pattern and used in the grid scenario.

Pattern 5: Cached Bitmap

Not much info on the MSDN (here). On the WPF forum archive (here) it’s explained that “The BitmapCache API is designed to cache your content (when rendering in hardware) in video memory, meaning it stays resident on your GPU. This saves you the cost of re-rendering that content when drawing it to the screen.” This seems like a great idea. I’m not sure, however, what are the pitfalls and how to use it.

Pattern 6: RenderTargetBitmap

The RenderTargetBitmap converts a Visual to a bitmap. I’m not sure if it’s relevant here. See here.

Edit: Regarding Paul Hoenecke question: I've written that "My application has to display many bitmap iages". I failed to mentions that I need to display about 800 images concurrently.

One can read about the performance issues involved at my SO questions WPF Bitmap performance and How can I make displaying images on WPF more “snappy”?

I've modified the description of pattern 1 to highlight the concept that the image controls aren't created or deleted (unless we want to display a larger or smaller grid). Only their Sources are set to different, new or null BitmapSources.

Edit: This question as posted on the WPF support forum, with some answers from MS personnel.

Answer 1

I'm unable to find a specific question in your post, other than asking for comments on the approaches below. I won't claim to know everything above but I'll tell you what I do know having worked for a while developing high-performance UIs using WPF and Silverlight.

Pattern 0: The Hack. Combining all into one image

I'd avoid this if possible. It sounds like you want to display a large wrap-panel of thousands of small images. Each image is therefore a thumbnail (as you cannot display 1000s of large images at once). As a result, I'd advocate caching/resize over combination.

Pattern 1: Reduce Image Size

If you are displaying 1,000 images on screen at once, consider the available screen real-estate. The average monitor is 1280x1024 pixels, or just over 1.3MPixels. 1000 images suggests you will get a maximum size of 1300 pixels per image, or 36*36. Lets say your images are 32*32 in size. You should definitely be creating a thumbnail of that image size to render on screen, then on click (or other action) show the full size image.

Also consider not only the render overhead of resizing a large image, but of sending a large image to the GPU to resize. That data requires bandwidth to send. A large image can be several megabytes whereas a thumbnail of size 32*32 could be a few kilobytes.

If you require dynamic sizing, fine, but you'll need to experiment with creating multiple thumbnails or generating them on the fly.

Pattern 2: Background pre-fetch

This is a technique I've not heard of, however it seems plausible. What is the overhead in your application? is it updating the Image.Source property or creating a new Image, tessellating, performing Layout and sending the information to render it to the GPU?

All the above occur on the CPU except for the final render. By reducing the overhead on the CPU side and updating the source you might be on to something. Combine this with WriteableBitmap as a source and you could further gain a performance improvement (see below).

Pattern 3: Drawing Context

Ok, all this does is allow you to queue up retained mode drawing calls using an "OnPaint" style syntax which is nothing like the old GDI OnPaint. In my experience OnRender doesn't improve performance, but it does allow for fine grained flexibility over what is drawn and when. OnRender provides you with a context, which has a DrawImage function, allowing a BitmapSource to be drawn to the rendering pipeline without the need for an Image control. This is good as it removes some overhead, however introduces problems similar to those seen in Pattern0 (you will lose layout and have to compute position of all your images). If you do this, you might as well invoke Pattern 0, which I advised against.

Pattern 4: Writeable Bitmaps

WriteableBitmaps are a little used and extraordinarily powerful subsystem within WPF. I use them to great effect to create a charting component capable of rendering large amounts of data in real-time. I would suggest checking out the WriteableBitmapEx codeplex project Disclosure, I have contributed to this once and seeing if you can combine it with other patterns. Specifically the Blit function which would let you write a cached bitmap to a bitmap source on an image.

For instance, a good technique might be Pattern 1 + 2 + 4.

You could have a grid of N Image controls on the screen at set locations in a grid control. Each of these is static and doesn't get scrolled out of view so there are no creations/deletions going on. Now, on top of this, resize your image and write to a WriteableBitmap which is set as the Source property on each image. As you scroll, get the next/previous thumbnails and update the sources using WriteableBitmapEx.Blit. Pow! virtualized, cached, multi-threaded imaging goodness.

Pattern 5: Cached Bitmap

This is an attempt by microsoft to do 1+2+4 as I discussed above. What it tries to do is after layout (CPU side), tessellation (CPU Side), sending retained mode render instructions to the GPU (CPU side) and rendering (GPU side) it caches a raster image of the rendered element which is re-used on the next rendering pass. Yes a little known fact about WPF is that wonderful GPU powered engine is terribly slow as it does most of its work on the CPU :P

I would experiment with BitmapCache and see how it performs. There are caveats, and they are that when you update your UIElement it has to recreate the cache so static elements will perform far better than dynamic. Also I've not seen a significant improvement in performance from using this whereas WriteableBitmap style techniques can give an order of magnitude improvement.

Pattern 6: RenderTargetBitmap

This final technique lets you render a UIElement to a bitmap - you know that - but what is interesting is this can perform a poor-mans thumbnail generator (or resize). For instance, set an Image with BitmapSource of your full size image. Now set the size of the Image control to 32*32 and render to bitmap. Voila! You have your BitmapSource thumbnail to use in conjunction with some swapping (Pattern 2) and/or writeable bitmaps.

Ok finally, just wanted to say the requirement you have will push WPF to its limits, however there are ways to get it to perform. Like I said, I have build systems which rendered thousands or millions of elements on the screen at once by using the wonderful workaround that is WriteableBitmap. Going down the standard WPF route will result in performance hell so you will have to do something exotic to solve this.

As I said my recommendation is 1+2+4. You must resize a thumbnail, of that I have no doubt. The idea of having a static grid of Image controls and updating the sources is very good. The idea of using WriteableBitmap (specifically WriteableBitmapEx blit function) to update the sources is also one worth exploring.

Good luck!