Fastest way to import millions of JSON documents to MongoDB

Question

I have more than 10 million JSON documents of the form :

["key": "val2", "key1" : "val", "{\"key\":\"val", \"key2\":\"val2"}"]

in one file.

Importing using JAVA Driver API took around 3 hours, while using the following function (importing one BSON at a time):

public static void importJSONFileToDBUsingJavaDriver(String pathToFile, DB db, String collectionName) {
    // open file
    FileInputStream fstream = null;
    try {
        fstream = new FileInputStream(pathToFile);
    } catch (FileNotFoundException e) {
        e.printStackTrace();
        System.out.println("file not exist, exiting");
        return;
    }
    BufferedReader br = new BufferedReader(new InputStreamReader(fstream));

    // read it line by line
    String strLine;
    DBCollection newColl =   db.getCollection(collectionName);
    try {
        while ((strLine = br.readLine()) != null) {
            // convert line by line to BSON
            DBObject bson = (DBObject) JSON.parse(JSONstr);
            // insert BSONs to database
            try {
                newColl.insert(bson);
            }
            catch (MongoException e) {
              // duplicate key
              e.printStackTrace();
            }


        }
        br.close();
    } catch (IOException e) {
        e.printStackTrace();  //To change body of catch statement use File | Settings | File Templates.
    }


}

Is there a faster way? Maybe, MongoDB settings may influence the insertion speed? (for, example adding key : "_id" which will function as index, so that MongoDB would not have to create artificial key and thus index for each document) or disable index creation at all at insertion. Thanks.

Answer 1

I'm sorry but you're all picking minor performance issues instead of the core one. Separating the logic from reading the file and inserting is a small gain. Loading the file in binary mode (via MMAP) is a small gain. Using mongo's bulk inserts is a big gain, but still no dice.

The whole performance bottleneck is the BSON bson = JSON.parse(line). Or in other words, the problem with the Java drivers is that they need a conversion from json to bson, and this code seems to be awfully slow or badly implemented. A full JSON (encode+decode) via JSON-simple or specially via JSON-smart is 100 times faster than the JSON.parse() command.

I know Stack Overflow is telling me right above this box that I should be answering the answer, which I'm not, but rest assured that I'm still looking for an answer for this problem. I can't believe all the talk about Mongo's performance and then this simple example code fails so miserably.

Answer 2

You can also remove all the indexes (except for the PK index, of course) and rebuild them after the import.