has anyone benchmarked Excelsior JET compile-to-native against a standard java runtime?

Question

Has anyone ever tried benchmarking a java application compiled to native using Excelsior JET against the same application running just-in-time on a normal java runtime? The only benchmark I can find is hosted on Excelsior's website and is for a single application; I'd like to see some independent results.

My application has both high CPU and memory usage (it's training machine learning models). I don't expect a performance increase with using Jet, but I may need to run on an environment that doesn't have the java runtime available (hence the compile to native) and I need to know if the performance would be much worse.

I know that Excelsior have an eval. available, but I'm hoping to save the time downloading, configuring, testing it etc...

Answer 1

DISCLAIMER: I work for Excelsior.

To answer your question, the most recent third-party benchmarking results that I know of are these, but I've just checked - that post is over four years old...

Now if you allow me to provide some advice:

Based on over 13 years of Excelsior JET market life, I can tell you that you have to test it against your particular application. No benchmark will give you any idea how your app will behave when natively compiled. As we say, your mileage will vary. We have customers reporting increases in speed that we've never seen in our lab, and we have prospects abandoning their evaluation early because their apps become much slower. (The latter often happens due to misconfiguration - e.g. not all classes get precompiled - so please contact our engineers if you run into something like that.)

There are also some case studies with performance comparisons on our Web site (#1, #2), but of course those are from users whose apps became faster. ;)

Update 15-Dec-2014: A word of caution: As of Excelsior JET 10, the 64-bit version is lagging behind the 32-bit one in terms of performance, as the former is based on the new, built from scratch compiler core that has less optimizations implemented. We are working hard on reversing this situation, but for now, if performance is important to you and you don't have any really good reason to use the 64-bit version, such as large heaps or the need to integrate with 64-bit native libraries, or the need to target OS X, stick with the tried and true 32-bit version.

Answer 2

I use Excelsior JET for a freeware game. In my tests, the game doesn't run faster or slower, but the framerate is way more consistent with Excelsior JET than with Oracle's VM. The startup of our game is way better with JET. JVM takes a while to warm up and load all classes, but with JET it's running immediately.

Memory consumption can be a bit higher with Excelsior JET if you use lots of threads - because by default Excelsior JET uses large fixed thread stack sizes. But that can be configured - so it's not an issue.

Support by email is also very good. You get to talk with engineers, not support personnel. If you do happen to find a bug a fix is usually only a day away. I'd also like to note that I had some weird unreproducable crash reports from users with earlier releases of JET (>2 years old or so). The current releases (7.6, and now 8.0) are both rock-solid. We don't see any crashes on ten-thousands machines.

In a nutshell: I can recommend Excelsior JET, performance is (in our case) as good as with the JVM.