Failing Android JUnit tests, not breaking my Ant script like I expect?

Question

Failing JUnit tests, not breaking my Ant script like I expect?

My continuous integration server runs an Ant script, which calls something like: /tests/ant run-tests

My JUnit tests run, but with errors: run-tests: [echo] run-tests-helper. [echo] Running tests ... [exec] [exec] com.zedray.stuff.FooBarTest:.... [exec] com.zedray.stuff.FooBarTest:.....INSTRUMENTATION_RESULT: shortMsg=Some error in your code. [exec] INSTRUMENTATION_RESULT: longMsg=java.security.InvalidParameterException: Some error in your code [exec] INSTRUMENTATION_CODE: 0

The errors are OK, but my build script keeps going (eventually publishing my broken app to my testers - bad!). What I would expect is for the instrimentaiton to throw a build error, so my continuous integration server (TeamCity in this case) realises that something has gone wrong and reports a broken build. The "failonerror" is already set in the relevant macrodef, so I'm not sure what else I can do?

/tests/build.xml

Running tests ...

Any ideas/suggestions on how to fix this?

Regards Mark

Answer 1

I did it another way, because I am using the ant test target that is in the Android build.xml file. This target prints to the standard out, so I captured stndout into a file then queried the file, using this result to fail my task.

 <target name="run-acceptance-tests" depends="clean, debug, install" >

    <property name="log.file" value="acceptance_tests_standard_out.txt" />
    <!-- because we don't have control over the 'test' target (to check for passes an fails) this prints to standard out
         we capture standard out into a file and query this to see if we have any test failures, using this to pass/fail our task -->
    <record name="${log.file}" action="start" />
    <antcall target="test" />
    <record name="${log.file}" action="stop" />

    <!-- do other stuff -->

    <loadfile property="tests.output" srcFile="${log.file}" />

    <echo>Checking for failures</echo>
    <fail message="acceptance tests failed!" >
        <condition>
            <contains string="${tests.output}" substring="FAILURES" />
        </condition>
    </fail>

    <echo>acceptance tests passed!</echo>
</target>

Answer 2

I had the same problem, and I ened up customize the "run-tests" target in my own build.xml like this, and there is no need to change the original android sdk test_rules.xml

<target name="run-tests" depends="-install-tested-project, install"
            description="Runs tests from the package defined in test.package property">
    <echo>Running tests ...</echo>
    <exec executable="${adb}" failonerror="true" outputproperty="tests.output">
        <arg value="shell" />
        <arg value="am" />
        <arg value="instrument" />
        <arg value="-w" />
        <arg value="-e" />
        <arg value="coverage" />
        <arg value="@{emma.enabled}" />
        <arg value="${manifest.package}/${test.runner}" />
    </exec>
    <echo message="${tests.output}"/>
    <fail message="Tests failed!!!">
         <condition>
             <contains string="${tests.output}" substring="FAILURES" />
         </condition>
    </fail>
</target>