Why is there a huge performance difference between 32 and 64 bit JDK? [duplicate]

Question

Recently I've been doing some benchmarking of the write performance of my company's database product, and I've found that simply switching to a 64bit JVM gives a consistent 20-30% performance increase.

I'm not allowed to go into much detail about our product, but basically it's a column-oriented DB, optimised for storing logs. The benchmark involves feeding it a few gigabytes of raw logs and timing how long it takes to analyse them and store them as structured data in the DB. The processing is very heavy on both CPU and I/O, although it's hard to say in what ratio.

A few notes about the setup:

Processor: Xeon E5640 2.66GHz (4 core) x 2
RAM: 24GB
Disk: 7200rpm, no RAID
OS: RHEL 6 64bit
Filesystem: Ext4
JVMs: 1.6.0_21 (32bit), 1.6.0_23 (64bit)
Max heap size (-Xmx): 512 MB (for both 32bit and 64bit JVMs)

Constants for both JVMs:

• Same OS (64bit RHEL)
• Same hardware (64bit CPU)
• Max heap size fixed to 512 MB (so the speed increase is not due to the 64bit JVM using a larger heap)

For simplicity I've turned off all multithreading options in our product, so pretty much all processing is happening in a single-threaded manner. (When I turned on multi-threading, of course the system got faster, but the ratio between 32bit and 64bit performance stayed about the same.)

So, my question is... Why would I see a 20-30% speed improvement when using a 64bit JVM? Has anybody seen similar results before?

My intuition up until now has been as follows:

• 64bit pointers are bigger, so the L1 and L2 caches overflow more easily, so performance on the 64bit JVM is worse.

• The JVM uses some fancy pointer compression tricks to alleviate the above problem as much as possible. Details on the Sun site here.

• The JVM is allowed to use more registers when running in 64bit mode, which speeds things up slightly.

Given the above three points, I would expect 64bit performance to be slightly slower, or approximately equal to, the 32bit JVM.

Any ideas? Thanks in advance.

Edit: Clarified some points about the benchmark environment.

Answer 1

From: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#64bit_performance

"Generally, the benefits of being able to address larger amounts of memory come with a small performance loss in 64-bit VMs versus running the same application on a 32-bit VM. This is due to the fact that every native pointer in the system takes up 8 bytes instead of 4. The loading of this extra data has an impact on memory usage which translates to slightly slower execution depending on how many pointers get loaded during the execution of your Java program. The good news is that with AMD64 and EM64T platforms running in 64-bit mode, the Java VM gets some additional registers which it can use to generate more efficient native instruction sequences. These extra registers increase performance to the point where there is often no performance loss at all when comparing 32 to 64-bit execution speed.
The performance difference comparing an application running on a 64-bit platform versus a 32-bit platform on SPARC is on the order of 10-20% degradation when you move to a 64-bit VM. On AMD64 and EM64T platforms this difference ranges from 0-15% depending on the amount of pointer accessing your application performs."

Answer 2

Without knowing your hardware I'm just taking some wild stabs

• Your specific CPU may be using microcode to 'emulate' some x86 instructions -- most notably the x87 ISA
• x64 uses sse math instead of x87 math, I've noticed a %10-%20 speedup of some math-heavy C++ apps in this case. Math differences could be the real killer if you're using strictfp.
• Memory. 64 bits gives you much more address space. Maybe the GC is a little less agressive on 64 bits mode because you have extra RAM.
• Is your OS is in 64b mode and running a 32b jvm via some wrapper utility?

Answer 3

The 64-bit instruction set has 8 more registers, this should make the code faster overall.

But, since processsors nowaday mostly wait for memory or disk, i suppose that either the memory subsystem or the disk i/o might be more efficient in 64-bit mode.