Simple and Scalable Non-Hosted alternative to Amazon S3

Question

I'm giving users the ability to attach images, videos, audio clips and other file attachments inside an existing web application. Some installs of our product have thousands of users so the volume of data will get very high.

Amazon S3 is the obvious solution but, due to legal reasons, cannot always be used. I need a solution that my customers can host themselves.

I'm therefore looking to build or adopt a file storage system with the following traits:

• Not hosted. Must be installable on my customers' Windows servers.
• Horizontally scalable to terabytes of storage.
• Similar operation to S3 such that I can make both approaches part of my product.
• Proven architecture

I've seen several suggestions for this on StackOverflow and other forums (Eucalyptus Walrus, Hadoop HDFS, MongoDB + GridFS, CouchDB, MogileFS) but couldn't find enough information to identify one as simple and proven.

I have experience with CouchDB and would jump on it if I could be sure that it'll do well with terabytes of video files but I haven't found a good success story to lean on.

Answer 1

The closest open source project is OpenStack swift (https://github.com/openstack/swift). It power's RackSpace CloudFiles. While S3 design is not exactly known OpenStack is built to give very similar functionality.
It is scale-out, have no single point of failure or bottlenecks.