MongoCursorNotFoundException -Query failed with error code -5

Question

We are getting the following exception.

com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 43249415092 not found on server xx.xx.xx.xx:27017' 
        at com.mongodb.connection.GetMoreProtocol.receiveMessage(GetMoreProtocol.java:115)
        at com.mongodb.connection.GetMoreProtocol.execute(GetMoreProtocol.java:68)
        at com.mongodb.connection.GetMoreProtocol.execute(GetMoreProtocol.java:37)
        at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:155)
        at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:219)
        at com.mongodb.connection.DefaultServerConnection.getMore(DefaultServerConnection.java:194)
        at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:197)
        at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:93)
        at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46)
        at com.mongodb.DBCursor.hasNext(DBCursor.java:152)

We are unable to find the root cause since we are getting this exception rarely.

We also observed that the application is unable to read from cursor but no exception is thrown.

In cases where no exception is thrown,we took the thread dump and found that the thread reading from mongo is in RUNNABLE state.

java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at com.mongodb.connection.SocketStream.read(SocketStream.java:85)
        at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:503)
        at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
        at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102)
        at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:416)
        at com.mongodb.connection.GetMoreProtocol.receiveMessage(GetMoreProtocol.java:112)
        at com.mongodb.connection.GetMoreProtocol.execute(GetMoreProtocol.java:68)
        at com.mongodb.connection.GetMoreProtocol.execute(GetMoreProtocol.java:37)
        at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:155)
        at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:219)
        at com.mongodb.connection.DefaultServerConnection.getMore(DefaultServerConnection.java:194)
        at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:197)
        at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:93)
        at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46)
        at com.mongodb.DBCursor.hasNext(DBCursor.java:152)

please help me in finding the root cause of this issue?

Answer 1

Recently, I met the same issue. After long time research, I figured out it. In my scenario, I have 4 mongos, which behind a load balance(return the mongos IP address randomly). In my connection string, I use the load balance host as the address of the mongoDB cluster .

When the app start, the mongoDB driver create a server with a connection pool. In the connection pool, there are mixed connections coming from 4 mongos.

When you query a large data(large than the batchSize), The first batch data comes from mongos A, then when the following batch request is pushed, the connection may connect to mongos B/C or D(source code). They can't find the cursor of course. So, the MongoCursorNotFoundException is thrown.

How to handle it?

Do not use balance host in your connection string. use all mongos IP address instead. Let mongoDB driver itself to balance the request.

WRONG: mongodb://your.load.balance.host:27000/yourDB?connectTimeoutMS=60000&minPoolSize=100&maxPoolSize=100&waitqueuemultiple=20&waitqueuetimeoutms=60000

RIGHT: mongodb://10.0.0.1:27017,10.0.0.2:27017,10.0.0.3:27017,10.0.0.4:27017/yourDB?connectTimeoutMS=60000&minPoolSize=100&maxPoolSize=100&waitqueuemultiple=20&waitqueuetimeoutms=60000

There is a better solution: You can configure a unique and dedicated host for each mongos, then modify the RIGHT connection string: replace the IP address by this host.