Is there any scalable open source file storage systems?

Question

It is interesting what software are using companies like dropbox, rapidshare, hotfile, and other to manage huge amount of files? Is there any open source system. I took a look on gridFS which is build on mongo-db. It looks like this is not the best choice because of speed (comparing to nginx). Or am i wrong?

I want system which can scale infinitely. By plugging servers in to the system. At list to 100 TB.

Answer 1

Checkout MogileFS - http://danga.com/mogilefs/ - an open source filesystem developed by Danga Interactive for using with their LiveJournal.com services.

If you don't want to/cannot use cloud services like Amazon's S3 and run your own servers, than MogileFS might be the right choice. MogileFS is reliable, management and scaling is very easy and cheap.

Unfortunately, I cannot provide any performance comparison or benchmarks to other filesystems. But you shouldn't expect MogileFS for the 1st place, cause it's working on application level, which can also be an advantage.

See also the Google Code page for more infos: http://code.google.com/p/mogilefs/

Answer 2

Dropbox is built on Amazon's S3 Dropbox - Where are my files stored.

You can find some open source options over here: Alternatives to Amazon S3

Answer 3

Hadoop's HDFS is a scalable file system. Another option is GlusterFS

Answer 4

GlusterFs is an open source distributed file system. Unlike HDFS it doesn't have a centralized metadata. Which means GlusterFS has no single point of failure.