rw_semaphore blocking I/O heavy threads

Question

I'm having some trouble tracing the performance bottleneck in my application. I'm doing computations on GPU and I'm using several threads to load and prepare the data. All files are stored on an SSD, but sometimes the performance is drastically reduced (vmstat shows that reads go from 300MB/s down to 30MB/s). When I ran ps I noticed that most worker threads are regularly blocked on call_rwsem_down_write_failed or on call_rwsem_down_read_failed. The actual output was:

-     - user   Dl   47.4 -               call_rwsem_down_read_failed
-     - user   Rl   48.5 -               -
-     - user   Dl   48.5 -               call_rwsem_down_write_failed
-     - user   Dl   47.2 -               call_rwsem_down_read_failed
-     - user   Dl   46.8 -               call_rwsem_down_write_failed
-     - user   Dl   49.1 -               call_rwsem_down_write_failed
-     - user   Dl   46.8 -               call_rwsem_down_write_failed
-     - user   Dl   47.2 -               call_rwsem_down_write_failed
-     - user   Dl   46.9 -               call_rwsem_down_write_failed
-     - user   Dl   49.3 -               call_rwsem_down_read_failed
-     - user   Dl   47.2 -               call_rwsem_down_write_failed
-     - user   Dl   48.4 -               call_rwsem_down_write_failed

What's strange is that the application is using Torch7 and libpng to load images, but neither of them uses rw_semaphores (I've greped the sources). Also, the threads don't perform any writes.

Since I can't find any place where they are used I suspect that it all happens in some syscall (can it be read?). Is there any way to check where these semaphores are used and eliminate the blocking?

Thanks in advance!

Answer 1

Is there any way to check where these semaphores are used …?

Provided that your kernel is suitably configured, you can use the system request w - Display list of blocked (D state) tasks - on the console or

echo w >/proc/sysrq-trigger
dmesg

to see the call stack traces.