c++ fork / std::thread and boost::timed_join

Question

I've got a program which is analysing a number of files (up to 10000). Platform is AMD64 with a unix operating system. Language is c++.

The program is currently forking the main process for every file (until a limit is reached, then it waits until a child is finished). The child is starting a boost::thread to execute the analysing functionality and then does a boost::timed_join on the boost::thread it created.

So, i've got some questions now.

1. Would it be reasonable to replace the fork with a more lightweight option? i'm not sure how much of a performance / memory gain this could be, i read somewhere on stackoverflow that fork isn't that expensive on unix.

2. How would i implement the killing of the thread which is doing the anlysis? start 2 threads, one executes the analysis and the other one is waiting and if the first one isn't finished after a certain amount of time, the second is killing the first? is there a more elegant way of doing this? if this is the option of choice, how do i kill another thread? getting the native handle and then pthread_kill() ?

3. if keeping the fork mechanism is adviceable: i've thought about replacing the boost::threads with std::threads, how would i replace the boost::timed_join ? letting the child process sleep for some time and then kill the thread would be a way of doing it but if the thread is finished before the time ends (which will happen all the time), the child would still be sleeping until the time is over -> overhead.

any advice would be appreciated!

Answer 1

I came across AFIO (docs, code) today. It is an aspiring Boost candidate built on ASIO and Boost.Thread.

AFIO is a linear scalable, batch, chainable, asynchronous closure execution engine extending ASIO and Boost.Thread specialised as a portable asynchronous file i/o implementation library.

This might streamline your async file processing.

Note that this would not fork other OS processes, but instead use threads or ASIO Proactor thread-like constructs instead. It will probably consume less system resources, but is more crash prone (as commented by @Ashalynd)