Java Performance Testing [duplicate]

Question

I want to do some timing tests on a Java application. This is what I am currently doing:

long startTime = System.currentTimeMillis();
doSomething();
long finishTime = System.currentTimeMillis();
System.out.println("That took: " + (finishTime - startTime) + " ms");

Is there anything "wrong" with performance testing like this? What is a better way?

Duplicate: Is stopwatch benchmarking acceptable?

Answer 1

The one flaw in that approach is that the "real" time doSomething() takes to execute can vary wildly depending on what other programs are running on the system and what its load is. This makes the performance measurement somewhat imprecise.

One more accurate way of tracking the time it takes to execute code, assuming the code is single-threaded, is to look at the CPU time consumed by the thread during the call. You can do this with the JMX classes; in particular, with ThreadMXBean. You can retrieve an instance of ThreadMXBean from java.lang.management.ManagementFactory, and, if your platform supports it (most do), use the getCurrentThreadCpuTime method in place of System.currentTimeMillis to do a similar test. Bear in mind that getCurrentThreadCpuTime reports time in nanoseconds, not milliseconds.

Here's a sample (Scala) method that could be used to perform a measurement:

def measureCpuTime(f: => Unit): java.time.Duration = {

  import java.lang.management.ManagementFactory.getThreadMXBean
  if (!getThreadMXBean.isThreadCpuTimeSupported)
    throw new UnsupportedOperationException(
      "JVM does not support measuring thread CPU-time")

  var finalCpuTime: Option[Long] = None
  val thread = new Thread {
    override def run(): Unit = {
      f
      finalCpuTime = Some(getThreadMXBean.getThreadCpuTime(
        Thread.currentThread.getId))
    }
  }
  thread.start()

  while (finalCpuTime.isEmpty && thread.isAlive) {
    Thread.sleep(100)
  }

  java.time.Duration.ofNanos(finalCpuTime.getOrElse {
    throw new Exception("Operation never returned, and the thread is dead " +
      "(perhaps an unhandled exception occurred)")
  })
}

(Feel free to translate the above to Java!)

This strategy isn't perfect, but it's less subject to variations in system load.

Answer 2

The code shown in the question is not a good performance measuring code:

1. The compiler might choose to optimize your code by reordering statements. Yes, it can do that. That means your entire test might fail. It can even choose to inline the method under test and reorder the measuring statements into the now-inlined code.

2. The hotspot might choose to reorder your statements, inline code, cache results, delay execution...

3. Even assuming the compiler/hotspot didn't trick you, what you measure is "wall time". What you should be measuring is CPU time (unless you use OS resources and want to include these as well or you measure lock contestation in a multi-threaded environment).

The solution? Use a real profiler. There are plenty around, both free profilers and demos / time-locked trials of commercials strength ones.

Answer 3

Using a Java Profiler is the best option and it will give you all the insight that you need into the code. viz Response Times, Thread CallTraces, Memory Utilisations, etc

I will suggest you JENSOR, an open source Java Profiler, for its ease-of-use and no overheads on CPU. You can download it, instrument the code and will get all the info you need about your code.

You can download it from: http://jensor.sourceforge.net/

Answer 4

Keep in mind that the resolution of System.currentTimeMillis() varies between different operating systems. I believe Windows is around 15 msec. So if your doSomething() runs faster than the time resolution, you'll get a delta of 0. You could run doSomething() in a loop multiple times, but then the JVM may optimize it.

Answer 5

Have you looked at the profiling tools in netbeans and eclipse. These tools give you a better handle on what is REALLY taking up all the time in your code. I have found problems that I did not realize by using these tools.