Can one read a remote file as an istream with libcurl?

Question

I'd like to use the libcurl library to open a remote date file and iterate through it with an istream. I've looked through the nice example in this thread but it writes the remote file to a local file. Instead I'd like to have the remote reads be pushed to an istream for subsequent programmatic manipulation. Is this possible? I would greatly appreciate help.

Best, Aaron

Answer 1

Boost's IO Stream might be a better solution than STL's own stream. At least it is much simpler to create a boost stream. From boost's own docs:

#include <curl/curl.h>
#include <boost/iostreams/stream.hpp>

class CURLDevice
{
    private:
        CURL* handle;
    public:
        typedef char                            char_type;
        typedef boost::iostreams::source_tag    category;

        CURLDevice()
        {
            handle = curl_easy_init();
        }

        CURLDevice(const std::string &url)
        {
            handle = curl_easy_init();
            open( url );
        }

        ~CURLDevice()
        {
            curl_easy_cleanup(handle);
        }

        void open(const std::string &url)
        {
            curl_easy_setopt(handle, CURLOPT_URL, url.c_str());
            curl_easy_setopt(handle, CURLOPT_CONNECT_ONLY, 1);
            curl_easy_perform(handle);
        }

        std::streamsize read(char* s, std::streamsize n)
        {
            size_t read;
            CURLcode ret = curl_easy_recv(handle, s, n, &read);
            if ( ret == CURLE_OK || ret == CURLE_AGAIN )
                return read;
            else
                return -1;
        }
};

typedef boost::iostreams::stream<CURLDevice> CURLStream;

int main(int argc, char **argv)
{
    curl_global_init(CURL_GLOBAL_ALL);

    {
        CURLStream stream("http://google.com");

        char buffer[256];
        int sz;

        do
        {
            sz = 256;
            stream.read( buffer, sz );
            sz = stream.gcount();
            std::cout.write( buffer, sz );
        }
        while( sz > 0 );
    }

    curl_global_cleanup();

    return 0;
}

Note: when I run the code above I get a segfault in CURL, this appears to be because I don't know exactly how to use curl itself.