Reusing Bitmap to Decode JPEG Data

Question

This part of my application displays a real-time image stream from an IP camera in a picture box. The stream is a typical motion JPEG formatted as a multi-part HTTP response.

All the code is written, my app works fine with very low CPU usage (1-2%) and a see-saw memory footprint (gradually increases until a GC is triggered then it's all recycled back down to normal). So this is more of an optimization question, best of practice etc, it's not required.

With that out of the way, right now I'm doing this in the normal Bitmap.FromStream() fashion, which generates a whole new bitmap for every frame, at 60Hz. Even if it's fine from a run-time perspective, that's jarring for me as a programmer.

If I were to use a more hands on approach, I'd preallocate the bitmap with known size (320x240 or 640x480, depending on options) and just decode the stream into my bitmap, but I don't see a .NET function that does this. I'd have to use my own JPEG decoder, which is both too much work, and more binary size (this thing is already 55mb of compiled code).

So my question is, am I missing something here? Is there a best way to do something like this? The function I mentioned would be perfect, but if not available, how else would I improve this?

Answer 1

Keep in mind that getting 2% cpu load from 60 fps jpegs is a very good result. Very little you can do to make that substantially better. Do make sure your benchmark is realistic, you cannot be doing much with the bitmap data when it gets this low. A potential trap is that the codec converts the compressed pixel data lazily. If you never actually, say, call Graphics.DrawImage() or Bitmap.LockBits() then the codec doesn't do anything more but parse the header. The .NET object you get is also very modest, just 20 bytes of GC heap is required.

There is fairly little reason to fear the underlying memory management. It is extremely opaque in GDI+, it completely takes care of it by itself without exposing anything more than a pointer. Image.FromFile() is by far superior, GDI+ uses a memory-mapped file to access the raw pixel data. No use for that in a camera app of course, Image.FromStream() is more involved due to the impedance mismatch between GC and native memory, there's a Marshal.Copy() under the hood to shovel bytes. The codec is likely to allocate itself to buffer/cache decoded pixels, you have no control over this whatsoever. No reason to fear this since you repeatedly use the exact same allocation, the Windows memory manager likes that a lot.

What you are asking for is possible. Two basic ways to do it. First one is the Bitmap constructor that takes an IntPtr, it needs to point to decoded pixel data. Which lets you re-use the raw pixel buffer. Second way is Bitmap.LockBits(), you can directly scribble into the pixel buffer that GDI+ allocated. There is no fundamental difference between the two.

You do need to bring your own JPEG decoder when you do this of course. The libjpeg library is the most commonly used one. Written in C, best way to incorporate it is with a C++/CLI wrapper. The libjpeg-turbo library is notable, promising 2-4x faster decodes. No real idea how these libraries compare to Microsoft's codec. Pay attention to the pixel format you decode to, 24bppRgb is a natural JPEG match so likely what you'll get out a codec but 32bppPArgb is by far the superior format (x10) if you display the images.