OpenMP overhead calculation

Question

Given n threads, is there a way that I can calculate the amount of overhead (e.g. # of cycles) that is required to implement a specific directive in OpenMP.

For example, given the code below

 #pragma omp parallel
 {
    #pragma omp for
    for( int i=0 ; i < m ; i++ )
       a[i] = b[i] + c[i];
 }

Can I calculate somehow how much overhead is required to create these threads?

Answer 1

I think the way to measure the overhead is to time both the serial and parallel versions, and then see how far off the parallel version is from its 'ideal' running time for your number of threads.

So for example, if your serial version takes 10 seconds and you have 4 threads on 4 cores, then your ideal running time is 2.5 seconds. If your OpenMP version takes 4 seconds, then your 'overhead' is 1.5 seconds. I put overhead in quotes because some of that will be thread creation and memory sharing (actual threading overhead), and some of that will just be unparallelized sections of code. I'm trying to think here in terms of Amdahl's Law.

For demonstration, here are two examples. They don't measure thread creation overhead, but they might show the difference between expected and achieved improvement. And while Mystical was right that the only real way to measure is to time it, even trivial examples like your for loop aren't necessarily memory bound. OpenMP does a lot of work that we don't see.

Serial (speedtest.cpp)

#include <iostream>

int main(int argc, char** argv) {
  const int SIZE = 100000000;
  int* a = new int[SIZE];
  int* b = new int[SIZE];
  int* c = new int[SIZE];

  for(int i = 0; i < SIZE; i++) {
    a[i] = b[i] * c[i] * 2;
  }

  std::cout << "a[" << (SIZE-1) << "]=" << a[SIZE-1] << std::endl;

  for(int i = 0; i < SIZE; i++) {
    a[i] = b[i] + c[i] + 1;
  }

  std::cout << "a[" << (SIZE-1) << "]=" << a[SIZE-1] << std::endl;

  delete[] a;
  delete[] b;
  delete[] c;

  return 0;
}

Parallel (omp_speedtest.cpp)

#include <omp.h>
#include <iostream>

int main(int argc, char** argv) {
  const int SIZE = 100000000;
  int* a = new int[SIZE];
  int* b = new int[SIZE];
  int* c = new int[SIZE];

  std::cout << "There are " << omp_get_num_procs() << " procs." << std::endl;

  #pragma omp parallel
  {
    #pragma omp for
    for(int i = 0; i < SIZE; i++) {
      a[i] = b[i] * c[i];
    }
  }

  std::cout << "a[" << (SIZE-1) << "]=" << a[SIZE-1] << std::endl;

  #pragma omp parallel
  {
    #pragma omp for
    for(int i = 0; i < SIZE; i++) {
      a[i] = b[i] + c[i] + 1;
    }
  }

  std::cout << "a[" << (SIZE-1) << "]=" << a[SIZE-1] << std::endl;

  delete[] a;
  delete[] b;
  delete[] c;

  return 0;
}

So I compiled these these with

g++ -O3 -o speedtest.exe speedtest.cpp
g++ -fopenmp -O3 -o omp_speedtest.exe omp_speedtest.cpp

And when I ran them

$ time ./speedtest.exe
a[99999999]=0
a[99999999]=1

real    0m1.379s
user    0m0.015s
sys     0m0.000s

$ time ./omp_speedtest.exe
There are 4 procs.
a[99999999]=0
a[99999999]=1

real    0m0.854s
user    0m0.015s
sys     0m0.015s