Sharing stdout among multiple threads/processes

Question

I have a linux program(the language doesn't matter) which prints it's log onto stdout. The log IS needed for monitoring of the process.

Now I'm going to parallelize it by fork'ing or using threads.

The problem: resulting stdout will contain unreadable mix of unrelated lines...

And finally The question: How would you re-construct the output logic for parallel processes ?

Answer 1

Sorry for answering myself...

The definite solution was to use the GNU parallel utility.

It came to replace the well known xargs utility, but runs the commands in parallel, separating the output into groups.

So I just left my simple one-process, one-thread utility as-is and piped its call through the parallel like that:

generate-argument-list | parallel < options > my-utility

This, depending on parallel's options can produce nicely grouped outputs for multiple calls of my-utility

Answer 2

If you're in C++ I'd consider using Pantheios or derivative version Boost::Log or using look at Logging In C++ : Part 2 or

If you're in another language then file locking around the IO operations is probably the way to go see File Locks, you can achieve the same results using semaphonres or any other process control system but for me file locks are the easiest.

You could also consider using syslog if this monitoring is considered as system wide.

Answer 3

Another approach, which we use, is to delegate a thread, logger thread, for logging. All other threads wishing to log will send it to logger thread. This method gives you flexibility as formatting of logs can be done is single place, which can also be configurable. If you don't want to worry about locks can use sockets for message passing.