Repair corrupted neo4j database (2.3.2 community)

Question

My neo4j-2.3.2-community database ran out of disk space (at 80G). I did a clean shutdown of it, then added more disk, and tried to start it back up, only to get cryptic error messages and have it refuse to start.

I shut it down this morning when I realized the disk was full and it wasn't accepting any more inserts. I didn't get back to trying to bring it back up until this evening. I apologize for the apparent gap in the logs between the shutdown and restart. The database is still in a prototype/proof-of-concept phase. I've been running jobs to load data in it for the past week or so. I really, really don't want to have to start over with that data loading process. It is way too slow as it is.

I'm hoping someone will tell me there is a "neo4j --force_repair" sort of command line option that will do the trick. My data loader is smart enough to check the last records it successfully inserted in the database before trying to load new records, so if we lose a few it should be ok.

Here is what I see in the console.log.

2016-01-31 16:25:56.648+0000 INFO  Successfully shutdown Neo4j Server
2016-01-31 16:25:56.810+0000 INFO  Successfully stopped database
2016-01-31 16:25:56.811+0000 INFO  Successfully shutdown database
2016-02-01 01:16:10.591+0000 INFO  Successfully shutdown Neo4j Server
2016-02-01 01:16:10.593+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
    at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
    at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
    at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
    at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
    ... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /home/neo4j/neo4j-community-2.3.2/data/graph.db
    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
    at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
    at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
    at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    ... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery@4ec23a22' failed to initialize. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
    at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
    at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
    ... 10 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=252, byteOffset=100663356} and entry version V2_1
    at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:207)
    at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:92)
    at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
    at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:77)
    at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
    at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:135)
    at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:72)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
    ... 19 more

Answer 1

This looks like the transaction logs have only been partially written. If that's true the datastore itself is unlikely to be corrupt.

WARNING: before doing any manual change in the data/graph.db folder be sure to have a backup copy in place.

Before proceeding, read the above paragraph. Backup taken? If yes, try to remove the transaction log files in data/graph.db by rm neostore.transaction.db.*. Try to start Neo4j. If it still fails to start, paste a snippet of data/graph.db/messages.log containing the latest startup sequence - this should give more insight.