Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to store a large directed unweighted graph with billions of nodes and vertices

The graph size is in the billions of nodes, and tens of billions of vertices.

It will store webpages urls, and links between webpages and it will be used for testing ranking algorithms.

Any language is fine but java is prefered.

Solutions i found so far:

  1. neo4j
  2. storing in sorted flat files

Yes, i have already read Best Way to Store/Access a Directed Graph.

Update

The data can be distributed on multiple computers and does not need to be fully in-memory.

like image 524
elhoim Avatar asked Nov 05 '22 19:11

elhoim


1 Answers

Depending on your implementation, another solution could be Terracotta. I think supports object graphs of this magnitude using a distributed virtual heap.

http://www.terracotta.org/web/display/docs/Concept+and+Architecture+Guide#ConceptandArchitectureGuide-VirtualHeap

like image 198
spieden Avatar answered Nov 15 '22 19:11

spieden