Java Realtime Performance

Question

I'm working with Java project that requires very advanced manipulations of images. In fact, I'm doing most of the manipulation using OpenCV, and I'm using JNI to wrap around the OpenCV functions that I need. I am extremely satisfied with the performance OpenCV gives, the people who wrote the OpenCV code deserve great great credit for the code. In sharp contrast to what I experience with the code Java devs wrote.

I started out optimistic over the choice of my programming language, my first working iteration of the project works fine, but its performance is nowhere near to realtime (getting about 1 frame per 2 seconds.) I've done some optimizations of MY code and its helped a lot. I've been able to push the frame rate up to about 10-20 frames per second, which is great, but what I'm finding is that to do any further optimizations I have to rewrite Java code to do the same thing but 10-20x more efficient.

I'm appalled at how the developers of Java pay very little attention to performance, especially when writing the classes for Media related classes. I've downloaded OpenJDK and I'm exploring the functions I'm using. For example, there is a function under the Raster class called getPixels(...) and it gets the pixels of the image. I was expecting this function to be a highly optimized function in the source code, with several calls to System.arrayCopy to further optimize performance. Instead what I found was extremely "Classy" code, where they are calling 5-6 different classes and 10-20 different methods just to accomplish what I can do in one line:

for (int i =0; i < n; i++) {
  long p = rawFrame[i];
  p = (p << 32) >>> 32;
  byte red = (byte) ((p >> 16) & 0xff);
  byte green = (byte) ((p >> 8) & 0xff);
  byte blue = (byte) ((p) & 0xff);
  byte val = (byte)(0.212671f * red + 0.715160f * green + 0.072169f * blue);
  data[i] = val;
  grayFrameData[i] = (val & 0x80) + (val & (0x7f)); 
}

The code above transforms an image to grayscale and gets the float pixel data, in roughly 1-10ms. If I wanted to do the same with Java built in functions, the conversion to grayscale itself takes 200-300ms and then grabbing the float pixels takes about 50-100ms. This is unacceptable for real time performance. Note to get a speedup, I make heavy use of bitwise operators, which Java devs shy away from.

I understand that they need to handle the general case, but even so, can't they at least give options for optimizations or at the very least a warning how slow this code may perform.

My question is, at this late point in the development (I already have my first iteration, not I'm working on a second that performs more in real time) should I bite the bullet and switch over to C/C++ where I can fine tune things a lot more, or should I stick with Java and hope things will become more realtime friendly so that I won't have to rewrite already implemented Java code to get a speedup.

I'm really beginning to become disgusted with how "classy" and slow Java really is. The amount of classes there are seems like overkill.

Answer 1

I've done computer vision work with Java, and I may get downvoted for saying this, but it is perfectly usable for computer vision and realtime stuff, you just have to know how to use it.

Potential Optimizations:

If you need help optimizing your code, I'd be glad to assist -- for example, I can tell you that you will probably get a performance boost by making a method

`public static final int getGrayScale(final int pixelRGB){
    return (0.212671f * ((pixelRGB >> 16) & 0xff) + 0.715160f * ((pixelRGB >> 8) & 0xff) + 0.072169f * ((pixelRGB) & 0xff));
}`

and using this in your for{pixels} loop. By using a method call, the JVM can much more heavily optimize this operation, and can probably optimize the for loop more too.

If you've got RAM to burn, you can create a static, final lookup table of output grayscale bytes for all possible 24-bit pixel pixel colors. This will be ~16 MB in RAM, but then you don't have to do any floating point arithmetic, just a single array access. This may be faster, depending on which JVM you are using, and whether or not it can optimize out array bounds checking.

Places to find similar, faster image processing code:

I would strongly suggest that you take a look at the code for the ImageJ image processing app (can't link thanks to StackOverflow being retarded) & its libraries, specifically ij.process.TypeConverter. Just like your code, it relies heavily on direct array operations with bit-twiddling and a minimum of extra array creation. The Java2D libraries (part of the standard JRE) and the Java Advanced Imaging(JAI) library (can't link thanks to StackOverflow being retarded) provide other ways to do image processing directly on image data rapidly without having to roll your own operation every time. For Java2D, you just have to be careful which functions you use.

Why the Java2D libraries are so indirect:

Most of the "class-iness" is due to supporting multiple color models and storage formats (I.E. HSB images, float-based color models, indexed color models). The indirection exists for a reason, and sometimes actually boosts performance -- the BufferedImage class (for example) hooks directly into graphics memory in recent VMs to make some operations MUCH faster. Indirection lets it mask this from the user a lot of the time.

Answer 2

My question is, at this late point in the development (I already have my first iteration, not I'm working on a second that performs more in real time) should I bite the bullet and switch over to C/C++ where I can fine tune things a lot more, or should I stick with Java and hope things will become more realtime friendly so that I won't have to rewrite already implemented Java code to get a speedup.

You are asking should I

1. Switch to a language where I can satisfy my performance requirements.
2. Stick with Java and hope things improve.

There might be other options.... but option 2 doesn't seem realistic, you can't just "hope" that the code becomes faster :p

A few points to note:

1. OpenJDK does not nescceraly have the same performance as the Sun JDK, have you tried the Sun JDK?
2. If the performance optimizations that you need done are in a few methods, then it might be worth re-writing them and sticking with Java...

Answer 3

My suggestion would be it depends on how important the manipulation of images is compared to the project as a whole, and relative to whatever advantages java does bring. Clearly you can write fast code in java (as you demonstrated) if you need to. However, if 80% of your project is going to consist of such optimization, I would certainly rethink Java as the language choice here.

On the other hand, if this represents 20% of the application, with the other 80% being the user functionality around providing this conversion, then perhaps having to do the work to get the manipulation done is a worthwhile trade off to not have to deal with your own memory management, and to have whatever other APIs java is giving you for user interaction (Web, Swing, SWT, whatever you are using).

Java is not known for its real time abilities due to the garbage collector. That might come to bite you as well, so be careful about that.

Answer 4

I don't know how much of a performance boost you'll get, but if you have a long-running process doing repetitive things you should try to run the Server Hotspot VM using java -server . It performs much better than the client VM that's the default on Windows, which is optimized for fast startup time.