Pthreads - High memory usage

Question

I am programming something in C that creates a lot of Pthreads in Linux on a 256Mb system. I usually have +200Mb free.

When I run the program with a low amount of threads it works, but once I make it create around 100 threads it gives errors because the system runs out of memory. I did several tests and each threads use almost 2Mb. The stack size of the threads is set to 16Kb.

The code I use to create each thread:

pthread_attr_t attr;
pthread_attr_init(&attr);
size_t stacksize;
stacksize = (double) 16*1024;
int res = pthread_attr_setstacksize (&attr, stacksize);
int res2 = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
if (res != 0 || res2 != 0) {
    logs << "pthread_attr_XX: error "+int2string(res);
    exit(-1);
}
pthread_t id;
pthread_create(&id, &attr, &Class::thread_callback, &some_var);

Is it normal or am I missing something? Thanks.

Answer 1

Not sure it will help, but try calling setrlimit with RLIMIT_STACK to limit the stack size to 16k before creating your first thread.

Answer 2

The system threads library is just not designed to support large numbers of threads on systems with very limited memory. You need to either use a threading library designed for that purpose or, better, use fewer threads.