InvalidRecordException from Neo4J after a JVM crash

Question

I'm running Neo4J 1.8 embedded, Java 6, on CentOS.

After a JVM crash I started getting a few exceptions of the following:

org.neo4j.graphdb.NotFoundException: Unable to load one or more relationships from Node[1169385]. This usually happens when relationships are deleted by someone else just as we are about to load them. Please try again.
        at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:530)
        at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:415)
        at org.neo4j.kernel.impl.core.NodeImpl.loadInitialRelationships(NodeImpl.java:368)
        at org.neo4j.kernel.impl.core.NodeImpl.ensureRelationshipMapNotNull(NodeImpl.java:345)
        at org.neo4j.kernel.impl.core.NodeImpl.getAllRelationshipsOfType(NodeImpl.java:195)
        at org.neo4j.kernel.impl.core.NodeImpl.getRelationships(NodeImpl.java:247)
        at org.neo4j.kernel.impl.core.NodeProxy.getRelationships(NodeProxy.java:92)
        at com.bizya.polosvc.neo4j.model.PersonImpl.getAllSelvesNoPropsLoad(PersonImpl.java:1303)
        at com.bizya.polosvc.neo4j.model.PathsFinderImpl.findPaths(PathsFinderImpl.java:189)
        at com.bizya.polosvc.neo4j.model.PathsFinderImpl.findPaths(PathsFinderImpl.java:49)
        at com.bizya.server.services.ReachabilityService.findReachabilities(ReachabilityService.java:82)
        at com.bizya.server.widget.BrowsingPageBuilder.buildAdditionalResultsModel(BrowsingPageBuilder.java:97)
        at com.bizya.server.widget.ContextBrowsingPageBuilder.buildAdditionalResultsModel(ContextBrowsingPageBuilder.java:81)
        at com.bizya.server.widget.ContextBrowsingPageBuilder.buildAdditionalResultsModel(ContextBrowsingPageBuilder.java:71)
        at com.bizya.server.widget.servlet.NetworkServlet.doGet(NetworkServlet.java:139)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at com.bizya.server.widget.servlet.MasterWidgetServlet.handleAuthorizedUser(MasterWidgetServlet.java:167)
        at com.bizya.server.widget.servlet.MasterWidgetServlet.service(MasterWidgetServlet.java:150)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:429)
        at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[1169385] is neither firstNode[969258] nor secondNode[1170375] for Relationship[3477951]
        at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:167)
        at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:105)
        at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
        at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:608)
        at org.neo4j.kernel.impl.core.NodeImpl.loadMoreRelationshipsFromNodeManager(NodeImpl.java:527)
        ... 38 more

I got a few more of those with different IDs.

These exceptions are consistent when trying to load particular nodes, I'm certain I wasn't deleting any relationships. I get the same error (org.neo4j.graphdb.NotFoundException: Unable to load one or more relationships ...) when inspecting these nodes in the command line shell.

It seems like a corruption of some sort. Any recommendation for how to fix it?

Answer 1

In light of what you have tried and your consistent troubles after a JVM crash, you may want to look into the possibility of a bug in Neo4J 1.8 that could be causing you the trouble. This could be caused by your relationship identifiers, like in this case. Unfortunately its a bug with the 1.8 version. You may want to consider upgrading to 1.9 or using the workaround suggested in the accepted answer from the link I posted.

Hope this points you in the right direction! And best of luck coding!