boost::thread_resource_error when more than 32705 threads

Question

I'm implementing a message passing algorithm. The messages propagate through the nodes of the graph, blocking until they have have received enough information (from other neighbours) to send a message.

The algorithm is easy to write if I put each message in its own thread and use a boost::condition to pause the thread until all the required information is available. I create many thousands of threads, but mostly only a few are active at any time. This seems to work pretty well.

My problem is, when unit testing I find that if I create more than about 32705 threads, I get

unknown location(0): fatal error in "Tree_test": std::exception: boost::thread_resource_error

and I don't know what causes this, or how to fix it.

There seems to be pleanty of memory available (Each thread only holds two pointers - the objects that the message passes between).

From this question: Maximum number of threads per process in Linux? I think the following information is relevent (although I don't really know what any of it means...)

~> cat /proc/sys/kernel/threads-max
1000000

(I increased this from 60120 - do I need to restart?)

 ~>ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I tried fiddling with the pending signals (my limit is very close to 2* that number) and stack size with ulimit -S -i 8191 - (I couldn't increase it) but these changes seemed to make no effect at all)

I'm on a 64 bit Ubuntu-10-10 if that helps...

Answer 1

I think with 32K threads on the system, you should look at potential solutions other than how to have more threads. For example, a thread pool (Boost has some things for this).

Anyway, on your system, aren't PIDs limited to 32768 or some such value? You're going to run out sooner or later, may as well design the system to allow processing more items than the max number of threads, I'd think.

That said, look at /proc/sys/kernel/pid_max to see your max PID--and try increasing it. This may get you beyond 32K (but may also cause unexpected behavior with programs not designed for unusually large PIDs, so be cautious).

And then you may be limited by the stack space (as opposed to virtual memory space). You could try creating threads with smaller stacks if you like.