Is it worth using a multithreaded blas implementation along with multiprocessing in Python?

Question

Suppose I have a 16 core machine, and an embarrassingly parallel program. I use lots of numpy dot products and addition of numpy arrays, and if I did not use multiprocessing it would be a no-brainer: Make sure numpy is built against a version of blas that uses multithreading. However, I am using multiprocessing, and all cores are working hard at all times. In this case, is there any benefit to be had from using a multithreading blas?

Most of the operations are (blas) type 1, some are type 2.

Answer 1

You might need to be a little careful about the assumption that your code is actually used multithreaded BLAS calls. Relatively few numpy operators actually use the underlying BLAS, and relatively few BLAS calls are actually multithreaded. numpy.dot uses either BLAS dot, gemv or gemm, depending on the operation, but of those, only gemm is usually multithreaded, because there is rarely any performance benefit for the O(N) and O(N^2) BLAS calls in doing so. If you are limiting yourself to Level 1 and Level 2 BLAS operations, I doubt you are actually using any multithreaded BLAS calls, even if you are using a numpy implementation built with a mulithreaded BLAS, like Atlas or MKL.

Answer 2

If you are already using multiprocessing, and all cores are at max load, then there will be very little, if any, benefit to adding threads that will be waiting around for a processor.

Depending on your algorithm and what you're doing, it may be more beneficial to use one type over the other, but that's very dependent.