How to run multiple QTest classes?

Question

I have a subproject where I put all my QTest unit tests and build a stand-alone test application that runs the tests (i.e. I run it from within Qt Creator). I have multiple test classes that I can execute with qExec(). However I don't know what is the proper way to execute multiple test classes.

Currently I do it in this way (MVCE):

tests.pro

QT -= gui
QT += core \
    testlib

CONFIG += console
CONFIG -= app_bundle
TEMPLATE = app
TARGET = testrunner

HEADERS += test_foo.h
SOURCES += main.cpp

main.cpp

#include <QtTest>
#include <QCoreApplication>
#include "test_foo.h"

int main(int argc, char** argv) {
    QCoreApplication app(argc, argv);

    TestFooClass testFoo;
    TestBarClass testBar;
    // NOTE THIS LINE IN PARTICULAR.
    return QTest::qExec(&testFoo, argc, argv) || QTest::qExec(&testBar, argc, argv);
}

test_foo.h

#include <QtTest>

class TestFooClass: public QObject
{
    Q_OBJECT
private slots:
    void test_func_foo() {};
};

class TestBarClass: public QObject
{
    Q_OBJECT
private slots:
    void test_func_bar() {};
};

However the documentation for qExec() says this is the wrong way:

For stand-alone test applications, this function should not be called more than once, as command-line options for logging test output to files and executing individual test functions will not behave correctly.

The other major downside is that there is no single summary for all the test classes, only for individual classes. This is a problem when I have dozens of classes that each have dozens of tests. To check if all tests passed I have to scroll up to see all the "Totals" of what passed/failed of each class, e.g.:

********* Start testing of TestFooClass *********
PASS   : TestFooClass::initTestCase()
PASS   : TestFooClass::test_func_foo()
PASS   : TestFooClass::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestFooClass *********
********* Start testing of TestBarClass *********
PASS   : TestBarClass::initTestCase()
PASS   : TestBarClass::test_func_bar()
PASS   : TestBarClass::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestBarClass *********

I'm also surprised my qExec() || qExec() works considering that the documentation says if a test failed qExec() returns a non-zero value, which should mean all the following qExec() calls wouldn't happen, but this seems not to be the case.

What is the proper way to run multiple test classes? And so that I can see at a glance if any of the hundreds of unit tests I have have failed.

Answer 1

I once found a nice solution using a plain Qt project (no TEMPLATE = subdirs) which uses a macro approach for creating the main function and automatic registering of all test classes (macro, too) with only a simple helper header file.

Here is a sample test class (only the relevant header file):

#ifndef FOOTESTS_H
#define FOOTESTS_H

#include "AutoTest.h"

class FooTests : public QObject
{
    Q_OBJECT
    private slots:
        void initTestCase();
        void test1();
        void test2();
        void cleanupTestCase();
};

DECLARE_TEST(FooTests)

#endif // FOOTESTS_H

and the main, which consumes every test class created this way:

#include "AutoTest.h"

TEST_MAIN

The code of AutoTest.h:

#ifndef AUTOTEST_H
#define AUTOTEST_H

#include <QTest>
#include <QList>
#include <QString>
#include <QSharedPointer>

namespace AutoTest
{
 typedef QList<QObject*> TestList;

 inline TestList& testList()
 {
  static TestList list;
  return list;
 }

 inline bool findObject(QObject* object)
 {
  TestList& list = testList();
  if (list.contains(object))
  {
   return true;
  }
  foreach (QObject* test, list)
  {
   if (test->objectName() == object->objectName())
   {
    return true;
   }
  }
  return false;
 }

 inline void addTest(QObject* object)
 {
  TestList& list = testList();
  if (!findObject(object))
  {
   list.append(object);
  }
 }

 inline int run(int argc, char *argv[])
 {
  int ret = 0;

  foreach (QObject* test, testList())
  {
   ret += QTest::qExec(test, argc, argv);
  }

  return ret;
 }
}

template <class T>
class Test
{
public:
 QSharedPointer<T> child;

 Test(const QString& name) : child(new T)
 {
  child->setObjectName(name);
  AutoTest::addTest(child.data());
 }
};

#define DECLARE_TEST(className) static Test<className> t(#className);

#define TEST_MAIN \
 int main(int argc, char *argv[]) \
 { \
  return AutoTest::run(argc, argv); \
 }

#endif // AUTOTEST_H

All credits goes to Rob Caldecott.