When does Neo4j release memory?

Question

I have a bit of confusion regarding memory management.

I am using neo4j in an embedded mode within a java application. Version - 1.8.1. JVM version - 1.6

I have a single threaded loader with a for loop that contains a nested for loop. Inside the external for loop, I do graphDb.beginTx();, get an id of a test object from mysql and create a Node for that. In the inner for loop, I am querying mysql to get a bunch of related objects. I create nodes for them and the corresponding relationship with the node that I created in the external loop.

At the end of the external for loop, I do tx.success() and tx.finish().

for (int x = 0; x < 10000000; x++) {
    Transaction tx = graphDb.beginTx();
    Node n = graphDb.createNode();
    Long id = n.getId();
    System.out.println("ID: " + id);
    n.setProperty("TestId", x); 
    * * * get the mysql ids * * * *
    for (int y = 0; y < mysqlidlist; y++) {
        Node n1 = graphDb.createNode();
        Long id2 = n1.getId();
        n1.setProperty("InnerTestId", y);
        Relationship rel = n.createRelationshipTo(n1, NodeRelation.ATTRIBUTE);
        rel.setProperty("Weight", 0.3);
    }
    tx.success();
    tx.finish();
    tx = null;
}

Based on my readings, I thought that Neo4j would then release the memory that the transaction had grabbed. However, I see the memory usage always increasing and after some time it hits the Xmx setting. I will profile it to make sure there are no other leaks. I am setting all the other variables to null at the end and that should help GC reap it in a better way.

Am I wrong in my understanding? If so, what is the recommended best practice on memory management?

Next step - the Single Threaded loader will be extended to be a multi-threaded one and as such, want to make sure that the memory management and transaction management is robust.

Thanks so much!

Regards,

Sachin

Answer 1

For this kind of massive insertion if you don't need transactions I suggest to use Batch Insertion.

http://docs.neo4j.org/chunked/milestone/batchinsert.html

---

Or, not recommended: If you want to use your snippet of code add something like: if(y%1000==0) { tx.success();tx.finish()} inside the for loop to commit and release the memory used every n inserts, but it will be slower