What is the best way to display millions of images in Java?

Question

You see that ?

Each brick of each house is an image of 16x16 pixels.

What you can see here a version based on simple JavaFX, with some Imageview moved on X and Y to give the effect of "construction".

I just adapt this to Swing using paintComponent.

The problem: - With JavaFX: my computer has trouble. What you see on the picture took 2 seconds to load and it is then moving very slow and jerky. - With Swing: I do not know how to adapt each block according to brightness, shadows, etc.. So it looks like this:

What method should I chose? Both have major drawbacks. I would have liked to keep the JavaFX method but also to find something else than Imageview. It should not be a good idea.

Answer 1

For information on Swing optimizations and implementations, see other answers as my answer is JavaFX specific.

Should you stick with a JavaFX implementation, here are a couple of things to try:

1. Use node.setCache(true) to switch on node caching.
2. Use node.setCacheHint(CacheHint.SPEED) to enable high speed node transforms.
3. Provide an implementation with no effects and another with effects and see if the one without effects performs noticeably better. (If that's the case you may need to use a more efficient effect chain or forgo some effects altogether).
4. Check the JavaFX system requirements to ensure that your setup meets the minimum requirements for hardware acceleration in terms of hardware/os/driver combinations.
5. If you are using Regions and css based processing be careful that the css system is not causing too much overhead (by, for example, having a version of the code which does not use css and comparing it's performance).
6. If needed, implement a level of detail zoom, (e.g. when zoomed out provide a single image for the entire house rather than a separate image for each house tile).
7. Make sure you are only loading a given Image once and reusing it in multiple ImageViews.
8. Use a profiler to identify bottlenecks.
9. Vary the test cpu or graphics card to see if either of those are a bottleneck.
10. Try a Java 8 preview, which features many internal performance optimizations for JavaFX.
11. If you are loading a lot of images, use background loading and have an animation or progress bar playing in front while the images are loading.
12. If you are loading lots of images, scale them upfront in the Image constructor so that they don't take up too much memory or require (potentially) extra processing power for the extra pixels (only really a consideration if you are using a lot of different very high resolution textures - which you don't seem to be doing).

For example:

Image tile = new Image("tile.png");

Group house = new Group();
house.setCache(true);
house.setCacheHint(CacheHint.SPEED);

Effect lighting = new Lighting();

for (int i = 0; i < houseWidth; i++) {
  // here is the critical part => don't do new ImageView(new Image("tile.png"))
  ImageView tileView = new ImageView(tile));
  tileView.setEffect(lighting);
  tileView.setCache(true);
  tileView.setCacheHint(CacheHint.SPEED);

  house.add(tileView);  
}

---

Mantrid's suggestions are interesting. I do believe that, with JavaFX you don't need to implement a dirty rectangle algorithm yourself (as the underlying platform is able to take care of dirty region processing for you). It may be that, because it is a generic mechanism, it isn't providing the level of optimization required for your specific case, in which case you need to handle the dirty processing yourself (e.g. by removing nodes from the scenegraph and re-adding them as appropriate).

Additionally, pre-calculating the blur/brightness/etc on the images could be done by defining effects in JavaFX, applying the effects to an offscreen ImageView node, then taking a snapshot of the offscreen ImageView node to get a pre-calculated image. This technique would allow you to reuse your existing JavaFX effects pipeline without the need to re-implement it using the ConvolveOp mechanism. Perhaps you may get the same performance level by just setting cache to true and cacheHint to speed on the ImageView nodes, as I believe this does a similar kind of thing behind the scene (i.e. increases speed at the cost of increased memory usage).

---

The JavaFX scenegraph is pretty efficient and can handle thousands of nodes. However, you may have more than that in your application. If the above optimization points don't help you, you may want to calculate your node count and post a reference to your question (and perhaps some source) to the open-jfx mailing list where the JavaFX developers who know the details of JavaFX application optimization are.

JavaFX 8 has far better support for 3D scenegraphs in than the (mostly useless) 3D scenegraph in JavaFX 2.2. It seems that your current application is psuedo-3d where you transform 2D objects, individually apply lighting effects and adjust the brightness of each tile to get a 3D look. If that's the case, using a fully hardware accelerated 3D scenegraph with a unified 3D lighting model may end up performing better, looking better and being easier to work with - you would have to evaluate it in the context of your application to see if it is worth switching your app from a 2D JavaFX scenegraph to a 3D JavaFX scenegraph or from JavaFX to Swing or some other tech like libgdx.

Addendum

Answers to some of Mizur's additional questions:

So why are they so power-consuming?

As can be seen from my answer, there are many aspects that go into performance, so singling out one particular reason as to why something is power-consuming is sometimes difficult and often not possible. Sometimes it is a combination of things which contribute to performance issues. The biggest wins usually come from optimizing stuff executed in inner most loops or by changing the strategy used to solve a problem.

If they need so much, please let me move tiles, or please just be displayed, no?

In terms of only consuming resources for the stuff you move, the JavaFX scenegraph has some optimizations for that and things like the cachehints can further improve performance for that. Perhaps they aren't enough for your particular use case or the way in which you are using the system. You may need to change your algorithms to reduce the system load.

Is there a way to generate images of each building with this technique of Imageview, then to integrate these images of buildings in the final scene?

Yes - you could render a building to an offscreen scene using many images and nodes, snapshot the offscreen scene to create a single image, then overlay that image over your final scene.

Answer 2

modifying blur/brightness/etc on the images before they are drawn

I haven't used it myself, but here's a hint - use Swing/AWT Kernel+ConvolveOp mechanism: http://www.java-tips.org/java-se-tips/java.awt.image/styling-digital-images-with-convolveop.html

BufferedImage biSrc = ...
BufferedImage biDest = ...

float[] data = new float[] { 0.0625f, 0.125f, 0.0625f, 0.125f, 0.25f, 0.125f, 0.0625f, 0.125f, 0.0625f };
Kernel kernel = new Kernel(3, 3, data);
ConvolveOp convolve = new ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP, null);
convolve.filter(biSrc, biDest);

operations are defined via a 3x3 matrix (data). you can do a search for examples for brigthness, blur etc.

the picture took 2 seconds to load but it is moving very slow and jerky after

use Dirty Rectangles algorithm: Dirty Rectangles