How is the current performance of the Mono virtual machine?

Question

The web is full of different kinds of performance test of different languages, compilers, and virtual machines. Yet hardly any of these test the performance using some real-world scenario. In addition, searching Google for these kind of benchmarks usually yields mainly only several years-old benchmarks as those have had the most time to gather links to them.

Do any of you have some real insight into the current performance of different virtual machines? In addition I'd especially like to know how Mono's performance compares to those of Microsoft's .Net and Sun's Java and how the performance of different virtual machines has evolved lately.

Answer 1

Measuring performance is a complicated matter. In the past, when languages were tested against the same operating system, running the same hardware and a very limited set of libraries it was possible to create benchmarks that could give a linear metric that would measure a system. It would let folks evaluate things from zero to ten, assimilate the result and move on quickly to the next subject.

Things have become more complicated with modern systems as there are multiple variables to take into account.

At least in Mono's case there are plenty of variables that come into play:

• Code:

  • The quality of the native code generated.
  • The speed at which the native code is generated.
  • The memory required to generate the code and to optimize the code
  • Is the code generator multi-threaded
  • Is the generated code thread-safe
  • Does it take advantage of CPU specific features, at compile time or JIT time.
  • Can it use SIMD instructions if available.
  • Does the language map itself neatly to multi-core platforms
  • Does the language provide enough parameters for an optimizier to tune your code automatically (Like Fortran does).

• Memory management:

  • The garbage collection algorithm used
  • Does the GC scale with multiple CPUs?
  • Is the GC incremental, or real time?
  • Does it support thread-local storage for improved performance?
  • Is it precise, compacting, generational, conservative and what mixes of each.

• API design:

  • Are the APIs designed for latency or bandwidth
  • Do APIs support automatically scaling to multiple CPUs.
  • Can you offload heavy duty work to a GPU?
  • Do your APIs support streaming interfaces

All of these things complicate matters very much and make a simple 0 to 10 answer very hard to give.

If you were to partition languages in classes, and you assume a competent and performance aware programmer, I would divide the world in these classes:

• Tier 1: Hand tuned assembly language by a professional
• Tier 2: Statically compiled, strongly typed languages: C/C++/Fortran/
• Tier 3: managed/JIT languages: Java/C#/.NET/Mono/Boo/F#
• Tier 4: dynamically typed/JITed languages: Google V8, IronPython, IronRuby
• Tier 5: pure interpreted languages: Python, Perl
• Tier 6: pure interpreted languages, with too many features for their own good.

But the languages do not paint an entire picture, the APIs that you will consume, the hosting operating system and other facilities will have a big impact on your results.

For example, recently in Mono we added support for replacing Mono's code gen engine with a more advanced, highly optimizing engine (the LLVM engine). It turns out that it was incredibly hard to find a test where the overhead of using LLVM was worth the extra memory use: desktop and web applications did not exhibit much of a difference. And this is probably due to the fact that these are mostly I/O bound applications.

Using LLVM was useful for scientific and computationally intensive applications, but in real life it did not make much of a difference from Mono's default optimization settings.

As for the specifics of Mono: although Mono does use Boehm's GC, what most folks do not realize is that Boehm can be configured in various ways. The default layman configuration is indeed not very powerful, but it works for everyone that wants a quick GC. Mono does not use Boehm in this mode, Mono configures Boehm extensively to work in precise-mode as well taking advantage of thread local storage, multi-core GC and release-memory-to-the-OS modes.

Answer 2

For a comparison of Java and Mono you can take a look at The Computer Language Benchmarks Game.