Why is there this Capacity Limit on Nodes and Relationships in neo4j?

Question

I wonder why neo4j has a Capacity Limit on Nodes and Relationships. The limit on Nodes and Relationships is 2^35 1 which is a "little" bit more then the "normal" 2^32 integer. Common SQL Databases for example mysql stores there primary key as int(2^32) or bigint(2^64)2. Can you explain me the advantages of this decision? In my opinion this is a key decision point when choosing a database.

Answer 1

It is an artificial limit. They are going to remove it in the not-too-distant future, although I haven't heard any official ETA.

Often enough, you run into hardware limits on a single machine before you actually hit this limit.

The current option is to manually shard your graphs to different machines. Not ideal for some use cases, but it works in other cases. In the future they'll have a way to shard data automatically--no ETA on that either.

Update: I've learned a bit more about neo4j storage internals. The reason the limits are what they are exactly, are because the id numbers are stored on disk as pointers in several places (node records, relationship records, etc.). To increase it by another power of 2, they'd need to increase 1 byte per node and 1 byte per relationship--it is currently packed as far as it will go without needing to use more bytes on disk. Learn more at this great blog post: http://digitalstain.blogspot.com/2010/10/neo4j-internals-file-storage.html

Update 2:
I've heard that in 2.1 they'll be increasing these limits to around another order of magnitude higher than they currently are.