How to perform multiple Guzzle requests at the same time?

Question

I can perform single requests using Guzzle and I'm very pleased with Guzzle's performance so far however, I read in the Guzzle API something about MultiCurl and Batching.

Could someone explain to me how to make multiple requests at the same time? Async if possible. I don't know if that is what they mean with MultiCurl. Sync would also be not a problem. I just want to do multiple requests at the same time or very close (short space of time).

Answer 1

An update related to the new GuzzleHttp guzzlehttp/guzzle

Concurrent/parallel calls are now run through a few different methods including Promises.. Concurrent Requests

The old way of passing a array of RequestInterfaces will not work anymore.

See example here

    $newClient = new  \GuzzleHttp\Client(['base_uri' => $base]);
    foreach($documents->documents as $doc){

        $params = [
            'language' =>'eng',
            'text' => $doc->summary,
            'apikey' => $key
        ];

        $requestArr[$doc->reference] = $newClient->getAsync( '/1/api/sync/analyze/v1?' . http_build_query( $params) );
    }

    $time_start = microtime(true);
    $responses = \GuzzleHttp\Promise\unwrap($requestArr); //$newClient->send( $requestArr );
    $time_end = microtime(true);
    $this->get('logger')->error(' NewsPerf Dev: took ' . ($time_end - $time_start) );

Update: As suggested in comments and asked by @sankalp-tambe, you can also use a different approach to avoid that a set of concurrent request with a failure will not return all the responses.

While the options suggested with Pool is feasible i still prefer promises.

An example with promises is to use settle and and wait methods instead of unwrap.

The difference from the example above would be

$responses = \GuzzleHttp\Promise\settle($requestArr)->wait(); 

I have created a full example below for reference on how to handle the $responses too.

require __DIR__ . '/vendor/autoload.php';
use GuzzleHttp\Client as GuzzleClient;
use GuzzleHttp\Promise as GuzzlePromise;

$client = new GuzzleClient(['timeout' => 12.0]); // see how i set a timeout
$requestPromises = [];
$sitesArray = SiteEntity->getAll(); // returns an array with objects that contain a domain

foreach ($sitesArray as $site) {
    $requestPromises[$site->getDomain()] = $client->getAsync('http://' . $site->getDomain());
}

$results = GuzzlePromise\settle($requestPromises)->wait();

foreach ($results as $domain => $result) {
    $site = $sitesArray[$domain];
    $this->logger->info('Crawler FetchHomePages: domain check ' . $domain);

    if ($result['state'] === 'fulfilled') {
        $response = $result['value'];
        if ($response->getStatusCode() == 200) {
            $site->setHtml($response->getBody());
        } else {
            $site->setHtml($response->getStatusCode());
        }
    } else if ($result['state'] === 'rejected') { 
        // notice that if call fails guzzle returns is as state rejected with a reason.

        $site->setHtml('ERR: ' . $result['reason']);
    } else {
        $site->setHtml('ERR: unknown exception ');
        $this->logger->err('Crawler FetchHomePages: unknown fetch fail domain: ' . $domain);
    }

    $this->entityManager->persist($site); // this is a call to Doctrines entity manager
}

This example code was originally posted here.

Answer 2

From the docs: http://guzzle3.readthedocs.org/http-client/client.html#sending-requests-in-parallel

For an easy to use solution that returns a hash of request objects mapping to a response or error, see http://guzzle3.readthedocs.org/batching/batching.html#batching

Short example:

<?php

$client->send(array(
    $client->get('http://www.example.com/foo'),
    $client->get('http://www.example.com/baz'),
    $client->get('http://www.example.com/bar')
));

Answer 3

Guzzle 6.0 has made sending multiple async requests very easy.

There are multiple ways to do it.

You can create the async requests and add the resultant promises to a single array, and get the result using the settle() method like this:

$promise1 = $client->getAsync('http://www.example.com/foo1');
$promise2 = $client->getAsync('http://www.example.com/foo2');
$promises = [$promise1, $promise2];

$results = GuzzleHttp\Promise\settle($promises)->wait();

You can now loop through these results and fetch the response using GuzzleHttpPromiseall or GuzzleHttpPromiseeach. Refer to this article for further details.

In case if you have an indeterminate number of requests to be sent(say 5 here), you can use GuzzleHttp/Pool::batch(). Here is an example:

$client = new Client();

// Create the requests
$requests = function ($total) use($client) {
    for ($i = 1; $i <= $total; $i++) {
        yield new Request('GET', 'http://www.example.com/foo' . $i);
    }
};

// Use the Pool::batch()
$pool_batch = Pool::batch($client, $requests(5));
foreach ($pool_batch as $pool => $res) {

    if ($res instanceof RequestException) {
        // Do sth
        continue;
    }

    // Do sth
}