Neo4J OutOfMemory error when querying

Question

I've recently started working with Neo4J and so far I haven't been able to find an answer to the problems I'm having, in particular with the server. I'm using version 1.8.1 and running the server as a service on Windows, not embedded. The graph I have has around 7m nodes and nearly 11m relationships.

With small queries and multiples of, things run nicely. However, when I'm trying to pull back more complex queries, potentially thousands of rows, things go sour. If I'm using the console, I'll get nothing and then after a few minutes or more the word undefined appears (it's trying to do something in Javascript but I'm not sure what). If I'm using Neo4JClient in .NET, it'll time out (I'm working this through a WCF service) and I suspect my problems are server side.

Here is a sample cypher query that has caused me problems in the console:

start begin = node:idx(ID="1234")
MATCH begin-[r1?:RELATED_TO]-n1-[r2?:RELATED_TO]-n2-[r3?:RELATED_TO]-n3-[r4?:RELATED_TO]-n4
RETURN begin.Title?, r1.RelationType?, n1.Title?, r2.RelationType?, n2.Title?, r3.RelationType?, n3.Title?, r4.RelationType?, n4.Title?;

I've looked through the logs and I'm receiving the following severe error:

SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuffer.append(Unknown Source)
    at java.io.StringWriter.write(Unknown Source)
    at java.io.PrintWriter.newLine(Unknown Source)
    at java.io.PrintWriter.println(Unknown Source)
    at java.io.PrintWriter.println(Unknown Source)
    at org.neo4j.cypher.PipeExecutionResult$$anonfun$dumpToString$1.apply(PipeExecutionResult.scala:96)
    at org.neo4j.cypher.PipeExecutionResult$$anonfun$dumpToString$1.apply(PipeExecutionResult.scala:96)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at org.neo4j.cypher.PipeExecutionResult.dumpToString(PipeExecutionResult.scala:96)
    at org.neo4j.cypher.PipeExecutionResult.dumpToString(PipeExecutionResult.scala:124)
    at org.neo4j.cypher.javacompat.ExecutionResult.toString(ExecutionResult.java:90)
    at org.neo4j.shell.kernel.apps.Start.exec(Start.java:72)
    at org.neo4j.shell.kernel.apps.ReadOnlyGraphDatabaseApp.execute(ReadOnlyGraphDatabaseApp.java:32)
    at org.neo4j.shell.impl.AbstractAppServer.interpretLine(AbstractAppServer.java:127)
    at org.neo4j.shell.kernel.GraphDatabaseShellServer.interpretLine(GraphDatabaseShellServer.java:92)
    at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:130)
    at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:114)
    at org.neo4j.server.webadmin.rest.ShellSession.evaluate(ShellSession.java:96)
    at org.neo4j.server.webadmin.rest.ConsoleService.exec(ConsoleService.java:123)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)

From an educated guess perspective looking at the stack trace, is it that I'm pulling back too many records? Since it's running out of memory whilst expanding the StringBuffer.

I've wondered whether GC could be playing a part, so I got hold of GCViewer. It doesn't seem to be Garbage Collection, I can add in a screenshot from GCViewer if you think it will be useful.

I've allocated the JVM anywhere between the default value and 8G of memory. Here are some of my settings from my configuration files (I'll try and include only the relevant ones). Let me know if you need any more.

Neo4J.properties

# Default values for the low-level graph engine
use_memory_mapped_buffers=false

# Keep logical logs, helps debugging but uses more disk space, enabled for legacy reasons
keep_logical_logs=true

Neo4J-server.properties

# HTTP logging is disabled. HTTP logging can be enabled by setting this property to 'true'.
org.neo4j.server.http.log.enabled=false 

Neo4J-Wrapper.conf (possibly inexpertly slotted together)

# Uncomment the following line to enable garbage collection logging
wrapper.java.additional.4=-Xloggc:data/log/neo4j-gc.log

# Setting a different Garbage Collector as recommended by Neo4J
wrapper.java.additional.5=-XX:+UseConcMarkSweepGC

# other beneficial settings that should boost performance
wrapper.java.additional.6=-d64
wrapper.java.additional.7=-server
wrapper.java.additional.8=-Xss1024k

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=1024

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=8000

Any help would be gratefully appreciated.

Answer 1

your query is simply too complex. when you have such a large graph than to be sure you will not reach your heap memory limit, you must have appropriate memory allocated. you might want to play a little bit with this configuration: io examples.

however, your query could be simplified to this:

start begin = node:idx(ID="1234")
MATCH p=begin-[r1:RELATED_TO*0..4]-n4
RETURN p

Answer 2

Craig the problem is that you use the Neo4j-Shell which is just an ops tool and just collects the data in memory before sending back, it was never meant to handle huge result sets.

You probably want to run your queries directly against the http endpoint with streaming enabled (X-Stream:true http-header) then you have no problem with that anymore.