GCC 4.5/Ubuntu 11.04 is auto-threading code?

Question

I apologize ahead of time that I don't quite have the proper jargon to describe my problem, and that I have likely not given enough information.

I've been running my MPI code under gcc 4.4 and OpenMPI/MPICH2 for months now with no issue on a variety of platforms. Recently I upgrade a set of servers and my desktop to Ubuntu 11.04 (running gcc 4.5 now) and ran an 8 task job on a node with 8 processors. Typically I see nearly 100% user CPU utilization, and now I see only 60% user CPU and over 30% system cpu. This leads to a remarkable slowdown of my code when run in this fashion.

Investigating further, I simply ran a serial job, and noted that the process reported 150+% cpu time was being used. So, my program was multithreading itself over many processors. I verified this explicitly using 'ps -eLF' and looking at the per-processor loads.

This is an incredibly bad and inefficient thing for my MPI code, and I have no idea where it comes from. Nothing has changed other than the move to Ubuntu 11.04 and gcc 4.5. I have verified this against varying OpenMPI versions.

I also moved binaries around between two binary-compatible machines. If I compile on another machine (ubuntu 10.10/gcc 4.4) and run there, everything is fine. Moving the binary to the Ubuntu 11.04 machine, the same binary begins threading itself.

It is worth noting that I have explicitly disabled all optimizations (-O0), thinking my default (-O3) could include something I didn't understand in 4.5. I get identical behavior regardless of the optimization level.

Please let me know what further information I can provide to determine the source of this problem.

* ADDITIONAL INFO *

Results of ldd in response to request. Simply, it's OpenMPI, libconfig, and scalapack, along with standard gcc stuff:

linux-vdso.so.1 =>  (0x00007ffffd95d000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2bd206a000)
libconfig.so.8 => /usr/lib/libconfig.so.8 (0x00007f2bd1e60000)
libscalapack-openmpi.so.1 => /usr/lib/libscalapack-openmpi.so.1 (0x00007f2bd151c000)
libmpi.so.0 => /usr/lib/libmpi.so.0 (0x00007f2bd126b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2bd0ed7000)
libblacsCinit-openmpi.so.1 => /usr/lib/libblacsCinit-openmpi.so.1 (0x00007f2bd0cd4000)
libblacs-openmpi.so.1 => /usr/lib/libblacs-openmpi.so.1 (0x00007f2bd0aa4000)
libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007f2bd022f000)
liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0x00007f2bcf639000)
libmpi_f77.so.0 => /usr/lib/libmpi_f77.so.0 (0x00007f2bcf406000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f2bcf122000)
libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007f2bceed3000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2bcecb5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2bd22fc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2bcea9f000)
libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007f2bce847000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2bce643000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2bce43f000)

All the best.

Answer 1

is it possible that you are running into this feature? http://gcc.gnu.org/onlinedocs/libstdc++/manual/parallel_mode.html

basically, certain standard library routines have parallel implementations.However, it is only turned on when the _GLIBCXX_PARALLEL macro is defined.

Answer 2

Seeing 60%/40% doesn't tell anything, perhaps the processing is just accounted differently. The only interesting figure here would be to compare the wallclock time of the total execution of your code.

Also, I would think that (if so) it is not your binary itself that is parallelized but the MPI libaries. To check that you would not only have to compile your code on the other machine but also to link it statically. Only then you can be sure that you run exactly the same binary code in all of its aspects on the other machine.

Then, also you can't be sure that the MPI library doesn't use C++ under the hood. I remember that it was quite difficult for one of the MPI libs (don't remember which) to convice to not to compile against the C++ interface, even if you were only doing C.