Aggregate data from several allure reports

Question

Allure report tool was designed to show one report per test run. Developers warned me about it. Anyway i need more from this convenient tool.

• Did you think to keep allure reports for several days/month?
• Did you think to analyze history data automatically?

Basically, I need to find answer on question: What tests are flaky for previous month and need to be stabilized? Unfortunately, I didn't find answer from the box on the main web site(http://allure.qatools.ru/) or in the internet.

My solutions:

Trying to solve first question: [Solved]

I tried to keep allure reports archived under jenkins CI tool, but i have to manage 1500 TestNg tests and allure report generation and archiving takes about 10min, which is unacceptable for our pipeline. Then i started to generate report on jenkins slave and ftp it to remote box according next structure: project_type/pipeline_type/suite_type/day/build_number.

I store reports as raw projects with btrfs compression, which allows to compress json files very effectively(files are compressed in 5 times) and files are accessible from ui.

UI is a django(simply love it) project, which follows files system structure.

This approach allows to aggregate data.

Trying to solve second question: [Not solved]

I need to aggregate data across all runs for suite_type(see structure above) and find out which tests are failed on main run or re-run attempts(we re-run our failed tests 3 times using FailedReporter TestNg listener) and match failed reasons with failed tests. Ideally i need to match failed tests results with stored reports.

For example:

We have 2 reports:

../parallel-suite/2015-10-21/560

../parallel-suite/2015-10-22/584

Each suite has failed test testCreditCard.

• Test failed in 560 build with reason - third party system was down.
• Test failed in 584 build on first re-run attempt with reason - third party system was down, on second re-run attempt with reason - IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck, on thrid attempt it passed.

As a result i need to know that third party system was down error was 2 times for 2 days and IndexOutOfBoundsException was 1 time for 2 days and test passed on second day on third re-run attempt.

Does someone work in this direction?

Answer 1

Yes, Allure supports aggregated reports. But it is more about:

• distributed execution (according to Allure model every test case can be labelled with host it was executed on)
• multiple executions (e.g. tests are written in different programming languages - you can't execute single Allure adapter).

As far as I know Allure will never support tracking historic test results, discovering trends etc. Historic test results might become enormous in amount and just crash your browser. Also timeline tab would make no sense any more. Other tabs would require significant changes. Why don't you use Sonar for those scenario?