How to ensure that we read all lines from boost::child process

Question

I saw the following code on boost::child documentation page where they explain how to read the output of a child process. http://www.boost.org/doc/libs/1_64_0/doc/html/boost_process/tutorial.html They say after running your child process, we can read it via this loop:-

bp::ipstream is; //reading pipe-stream
bp::child c(bp::search_patk("nm"), file, bp::std_out > is);

//then later
while (c.running() && std::getline(is, line) && !line.empty())
        data.push_back(line);

I have 2 questions here :-

1. If c.running() returns false, we simply exit the loop. And in that case, the stream is above may still carry data which gets lost ?
2. What is the best way to read stdout and stderr both, while making sure the process exit() doesn't create a deadlock There is a warning on the page saying :-
   The pipe will cause a deadlock if you try to read after nm exited

I wish to capture both stdout and stderr without having to worry about nm has exited or not above.

Answer 1

I think there's no proper way unless you use asynchronous methods.

Perhaps you can simple get a future to a vector and use string_views into that if you somehow really need it line-by-line.

std::future<std::vector<char> > output, error;

boost::asio::io_service svc;
bp::child c(bp::search_path("nm"), file, bp::std_out > output, bp::std_err > error, svc);
svc.run();

To read exactly like you did before you can use an istream on top of the vector:

#include <boost/process.hpp>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream_buffer.hpp>
#include <iostream>

namespace bp = boost::process;
namespace bio = boost::iostreams;
std::string const file = "./a.out";

int main() {
    std::future<std::vector<char> > output, error;

    boost::asio::io_service svc;
    bp::child c(bp::search_path("nm"), file, bp::std_out > output, bp::std_err > error, svc);
    svc.run();

    //then later
    {
        auto raw = output.get();
        std::vector<std::string> data;
        std::string line;
        bio::stream_buffer<bio::array_source> sb(raw.data(), raw.size());
        std::istream is(&sb);

        while (std::getline(is, line) && !line.empty())
            data.push_back(line);

        std::cout << data.at(rand()%data.size()) << "\n";
    }

}

Answer 2

I had the same issue... The best way to deal with this is by using async i/o.

Unfortunately, the boost documentation @ http://www.boost.org/doc/libs/master/doc/html/boost_process/extend.html#boost_process.extend.async was wrong... It makes it all look simple, but doesn't show that the buffers must be sized beforehand, and glosses over many details.

Here's my function, which send or receives one buffer in one go (no interaction like question/response0, I use stderr for error checking, since that's what was needed for my app, but you could catch the app's exit code by calling 'c.exit_code();`.

using tstring=basic_string<TCHAR>;

void Run(
    const tstring& exeName;
    const tstring& args,
    const std::string& input,
    std::string& output,
    std::string& error
)
{
    using namespace boost;

    asio::io_service ios;

    std::vector<char> vOut(128 << 10);
    auto outBuffer{ asio::buffer(vOut) };
    process::async_pipe pipeOut(ios);

    std::function<void(const system::error_code & ec, std::size_t n)> onStdOut;
    onStdOut = [&](const system::error_code & ec, size_t n)
    {
        output.reserve(output.size() + n);
        output.insert(output.end(), vOut.begin(), vOut.begin() + n);
        if (!ec)
        {
            asio::async_read(pipeOut, outBuffer, onStdOut);
        }
    };

    std::vector<char> vErr(128 << 10);
    auto errBuffer{ asio::buffer(vErr) };
    process::async_pipe pipeErr(ios);
    std::function<void(const system::error_code & ec, std::size_t n)> onStdErr;
    onStdErr = [&](const system::error_code & ec, size_t n)
    {
        error.reserve(error.size() + n);
        error.insert(error.end(), vErr.begin(), vErr.begin() + n);
        if (!ec)
        {
            asio::async_read(pipeErr, errBuffer, onStdErr);
        }
    };

    auto inBuffer{ asio::buffer(input) };
    process::async_pipe pipeIn(ios);

    process::child c(
        exeName + _T(" ") + args, 
        process::std_out > pipeOut, 
        process::std_err > pipeErr, 
        process::std_in < pipeIn
    );


    asio::async_write(pipeIn, inBuffer, 
        [&](const system::error_code & ec, std::size_t n) 
        {
            pipeIn.async_close();
        });

    asio::async_read(pipeOut, outBuffer, onStdOut);
    asio::async_read(pipeErr, errBuffer, onStdErr);

    ios.run();
    c.wait();
}

The app I execute is a decoder/encoder, I send an entire file to process, and receive the results simultaneously this way. No file size limit.

IMPORTANT

There is a bug fix in boost 1.64, only affects Windows, apparently

in file boost\process\detail\windows\async_pipe.hpp: ref: https://github.com/klemens-morgenstern/boost-process/issues/90

line 79:

    ~async_pipe()
    {
//fix
        //if (_sink .native()  != ::boost::detail::winapi::INVALID_HANDLE_VALUE_)
        //    ::boost::detail::winapi::CloseHandle(_sink.native());
        //if (_source.native() != ::boost::detail::winapi::INVALID_HANDLE_VALUE_)
        //    ::boost::detail::winapi::CloseHandle(_source.native());
        boost::system::error_code ec;
        close(ec);
//fix
    }