PHP loses mongoDB cursor despite long timeouts

Question

I'm running a long mongoDB query like this:

foreach($xyz->find(...)->timeout(24 * 60 * 60 * 1000)->maxTimeMS(24 * 60 * 60 * 1000) as $document) {
    ...
}

But despite those 24 hour timeouts for the client and the server, the script exits with a MongoCursorException after a few minutes:

localhost:27017: could not find cursor over collection xyz

I'm on PHP 5.4 with the v1.6.10 mongoDB driver. The DB is mongoDB 3.0.4. PHP will connect to a mongos instance, the collection xyz is sharded.

Any idea what may cause this exception?

Answer 1

I have to say, I have similar experience, I have a collection which I look trough via

$items  = $col -> find(['data' => 'OK']);
$items->timeout(-1);
$items->maxTimeMS(3600*1000);

but after walk trough via

foreach($items as $item)
{
   ///... processing
}

after about 12 - 15 minutes, I get the same error

could not find cursor over collection

In the collection, there is about 150000 records, and the interesting fact is, that the error always appears after processing 123479 records, regardless the content of the 123480th record and regardless the time spent by the processing of the respective records (time necessary for processing the record can vary depending on the content).

As I cannot find any cause of the error, after several tries to improve mongo settings, including upgrade of driver for PHP, I am now doing whole process in batches, which prevents losing the cursor. It works fine, however, I would like to know the more clean solution as well.

Answer 2

It appears that there are more timeouts which are not all supported by PHP (yet). One of them is maxIdleTimeMS:

The maximum number of milliseconds that a connection can remain idle in the pool before being removed and closed.

What happened in my case was that, due to the database being busy at times, this long running query was idle for too long so its cursor got dropped server-side.

Setting this parameter would've helped but since it's not available, I've found that what solved the problem was to decrease the batch size:

$cursor->batchSize(-20)

In this example, at most 20 documents are transferred between the server and the client for each batch. This reduces the probability that the connection is idle for too long. Of course, the exact value depends on the load of your database, how long you need to process each document etc.