Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to run Postgres tests in parallel with NodeJS, Jest and Knex?

I have a project that was developed using Postgres in production/staging and Sqlite in development. With Sqlite we were able to run all our tests in parallel in 13 seconds.

This was a great strategy for the initial development, but there are some things that Sqlite just can't do that we need (Eg. dropping columns and adding new foreign keys). So I'm thinking that we should drop Sqlite and just use Postgres.

The test suite takes about a minute to run and if a test fails I routinely have to manually remove the migration tables. It doesn't provide a good testing experience.

Do you have any suggestions for running tests on a Postgres database in parallel using NodeJS, Knex and Jest?

like image 959
user3654410 Avatar asked Nov 15 '17 14:11

user3654410


1 Answers

Running migrations and rolling them back before every test is really slow, it might take even seconds to run. So you might not need to be able to run tests parallel to reach good enough speed.

If you setup your tests in a way that you just drop / create / migrate once before starting to run tests and between tests just truncate all data from the tables and populate it with fresh data it should be 10x faster (truncate takes usually around less than 50ms). You can truncate all the tables easily for example with knex-db-manager package.

If you really like to run postgresql tests parallel, you need as many test databases that you are running parallel tests. You could create "pool" of test databases (testdb-1, testdb-2, testdb-3,...) and in each jest test you first have to request database from your test database pool so that you can really run multiple tests at the same time and they wont be interfering with the same database.

Lastly one more pretty fast method to reset data in test database is to use pg-dump / pg-restore and binary db dumps. It might be faster or easier to handle in some cases than just running population scripts. Especially in case, where you are using the same initial data in every test.

In this way you create initial state for the test database before starting to run tests and take dump of it. For dumping and restoring I wrote these small helpers, which I might add to knex-db-manager at some point.

Parameter for dumping / restoring are a bit tricky (specially setting the password) so these snippets might help:

Dumping:

      shelljs.env.PGPASSWORD = config.knex.connection.password;
      const leCommand = [
        `pg_dump -a -O -x -F c`,
        `-f '${dumpFileName}'`,
        `-d ${config.knex.connection.database}`,
        `-h ${config.knex.connection.host}`,
        `-p ${config.knex.connection.port}`,
        `-U ${config.knex.connection.user}`,
      ].join(' ');

      console.log('>>>>>>>> Command started:', leCommand);
      shelljs.rm('-f', dumpFileName);
      shelljs.exec(leCommand, (code, stdout, stderr) => {
        console.log('======= Command ready:', leCommand, 'with exit code:', code);
        if (code === 0) {
          console.log('dump ready:', stdout);
        } else {
          console.log('dump failed:', stderr);
        }
      });

Restore:

  shelljs.env.PGPASSWORD = config.knex.connection.password;
  const leCommand = [
    `pg_restore -a -O -x -F c`,
    `-d ${config.knex.connection.database}`,
    `-h ${config.knex.connection.host}`,
    `-p ${config.knex.connection.port}`,
    `-U ${config.knex.connection.user}`,
    `--disable-triggers`,
    `'${dumpFileName}'`,
  ].join(' ');

  console.log('>>>>>>>> Command started:', leCommand);
  shelljs.exec(leCommand, (code, stdout, stderr) => {
    console.log('======= Command ready:', leCommand, 'with exit code:', code);
    if (code === 0) {
      console.log('restore ready:', stdout);
    } else {
      console.log('restore failed:', stderr);
    }
  });
like image 132
Mikael Lepistö Avatar answered Oct 27 '22 00:10

Mikael Lepistö