Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Error 10334 "BSONObj size is invalid" when importing a back up with mongorestore

I have a MongoDB collection backup containing many small documents. The back up was produced by mongodump, but when I try to import it using mongorestore I get an error:

AssertionException handling request, closing client connection: 10334 BSONObj size: 18039019 (0x11340EB) is invalid. Size must be between 0 and 16793600(16MB)

I'm running MongoDB version 3.0.3 (from trunk).

like image 750
Jonathan Lee Avatar asked Jun 01 '15 05:06

Jonathan Lee


3 Answers

Using --batchSize=100 fixes this issue for me every time.

e.g. mongorestore -d my-database --batchSize=100 ./database-dump-directory

like image 151
Stefan Liedle Avatar answered Sep 20 '22 00:09

Stefan Liedle


Basically mongoDB accept the size of the document should be less than 16MB.If you intent to use the document which is more than 16MB ,you can use gridfs. Each document takes memory power of 2 size allocation. Your application should ensure the size of bson document it is generating..Or else you can use the different data model rather than embedding all the data in one doc.

like image 22
Amaresh Avatar answered Sep 22 '22 00:09

Amaresh


mongorestore send insert commands by batch in a {"applyOps", entries} document. This document is (AFAIK) limited to 16MB just like any other document.

According to the sources there are "pathological cases where the array overhead of many small operations can overflow the maximum command size". The variable oplogMaxCommandSize is used to help mongorestore to not fail on such cases. It was raised to 16.5M at some point during the 3.0... development. That was too optimistic. It was lowered back to 8M later (JIRA TOOLS-754).

If you need to, you may adjust that value yourself according to your needs. And then recompile the tools.

like image 25
Sylvain Leroux Avatar answered Sep 18 '22 00:09

Sylvain Leroux