Compiling Python from source: multiple threads for tests?

Question

I compile Python 3.6 from sources using:

./configure --enable-optimizations --prefix=/local/py/path
make -j8

The compilation itself runs on 8 threads as expected when passing -j8 to make. However, since I also use --enable-optimizations I read that skipping tests is not recommended, but the tests run sequentially and only on one thread leading to very long compilation time.

Is there something I could do to let the tests be executed on multiple threads?

Answer 1

I do not build Python often enough to prompt me to have a closer look (once or twice after each release tops), but you've given me an extra impulse and I've poked around a bit.

I presume you're referring to (currently) 406 tests executed as part of building the default target. That would mean the run_profile_task target of its Makefile. Which runs python -m test.regrtest --pgo with your freshly built interpreter as defined in the PROFILE_TASK make variable.

Long story short, instead of of:

make -j8

You can run:

make PROFILE_TASK="-m test.regrtest --pgo -j8" -j8

Which just repeats the default definition (as of now) and appends -j8 which conveniently is the same as the one used by make.

And since I am a curious person, I had to try comparing both. On the machine I currently got booted up (not a very beefy laptop) it reduced regression test times from ~23 min to ~8 min. So, thanks for an excuse to look around, this may come in handy. :)