What maven plugin is to be used for JMeter? jmeter-maven-plugin or chronos-jmeter-maven-plugin?

Question

I need to setup performance tests which are run automatically triggered by a CI system. For that I want to use JMeter due to some scripts and experience already exist and I want to combine it with Maven.

During my research for a reasonable plugin I found that two plugins are existing:

1. jmeter-maven-plugin: http://wiki.apache.org/jmeter/JMeterMavenPlugin
2. chronos-jmeter-maven-plugin: http://mojo.codehaus.org/chronos/chronos-jmeter-maven-plugin/usage.html

Which one is better to be used? Both seem to be currently maintained and under development. Is there any experience on this? Even the configuration is similar.

I would be happy to get some hints to help me descide without playing around with both plugins for some days.

Answer 1

I haven't yet used the .jmx files with maven and specifically those plugins you mention.

But I can think of a way how to do it if I needed that.

So consider this, you can execute jmeter test in no gui mode.

1. Create a shell script wrapper that will execute the jmeter test in no gui mode, example (jmeter_exe.sh):

$JMETER_HOME/bin/jmeter.sh -n -t MY_LOAD_TEST.jmx -l resultFile.jtl

So this will execute the given script and store results in the .jtl file, you can use that to display your test results maybe this post will be useful to you, it's off topic for now.

With step one done.

2.You could then create directory scripts in your project root. Than you can put this in your pom.xml :

<plugin>
  <artifactId>exec-maven-plugin</artifactId>
  <groupId>org.codehaus.mojo</groupId>
  <executions>
    <execution>
      <id>Run load Test</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <executable>${basedir}/scripts/jmeter_exe.sh</executable>
      </configuration>
    </execution>
  </executions>
</plugin>

And voila your test is executed during generate-sources phase. This might have been easier with the plugins you mentioned but I have no knowledge of those, this is what just came to my mind.

Answer 2

Use jmeter-maven-plugin: http://wiki.apache.org/jmeter/JMeterMavenPlugin.

It's the de-facto one and (as @Ardesco mentioned above) it doesn't require anything to be installed, which gives you abstraction on where JMeter executable is installed and all those kind of problems...

Answer 3

Word(s) of warning on the apache plugin (lazerycode):

• It suppresses JMeter output by default, add the following configuration settings to prevent that:

<configuration>  
    <suppressJMeterOutput>false</suppressJMeterOutput>  
    <!-- to override debug logging from the plugin (although also in jmeter.properties) -->  
    <overrideRootLogLevel>debug</overrideRootLogLevel>  
    <jmeterLogLevel>DEBUG</jmeterLogLevel>  
</configuration>

• Looking at the source (of version 1.8.1), it seems the -Xms and Xmx are limited to 512

• The plugin swallows exceptions so your tests may fail but you don't know why. It looks like they've just completed but not provided results.

• The jmeter mojo kicks off jmeter as a new java process but does not provide the capacity to provide any arguments to this execution. So if exceptions are swallowed (See above), and logging isn't sufficient (which it may not be) it's not easy to debug the process to fing out what's wrong. We (my colleague) added the debug args to the process execution and debugged the jmeter call to find out.

• you get informative output running jmeter directly for dev purposes. I'd say it's even more informative in the jmeter UI output.

I've not used chronos mind.