neo4j "empty" database takes up a lot of disk space

Question

I've inserted ~2M nodes (via Java API), and deleted them after a day or two of usage (through java too). Now my db has got 16k nodes, and weights 6 GB.

Why this space wasn't freed?

What may be the cause?

Answer 1

The data/graph.db directory contains multiple items:

• Store itself, split into multiple files
• Indexes
• Transaction log files
• Log files (messages.log)

All your operations are stored in the transaction logs and then expire according to the keep_logical_logs setting. Not sure what the default value is, by I presume that you might have quite some space in use there.

I'd suggest to check what is taking up the space.

Also, we have sometimes seen that the space in use (reported with du for example) differs when Neo4j is running and stopped.

Answer 2

In addition to Alberto's answer, the store is not compacted. It leaves the empty records for reuse, and they will stay there forever. As far as I know, there is no available tool to compact the store (I've considered writing one myself, but usually convince myself that there aren't that many use cases affected by this).

If you do have a lot of churn where you are inserting and deleting records often, it's a good idea to restart your database often so it will reuse the records that it has marked as deleted.

As Alberto mentions, one of the first things I set (the other being the heap size) when I install a new neo4j is the keep_logical_logs to something like 1-7 days. If you let them grow forever (the default), they will get quite large.