Is there a server that provides an Amazon S3 style API locally? [closed]

Question

We make distributal software that stores some data (attachments) in a) a database or b) Amazon S3. The database is used because it requires no other configuration. Amazon S3 is the better option.

What we want now is a solution for customers that don't want to use Amazon S3. We can obviously just use the filesystem but this can be problematic if there are multiple web servers and the files need to be replicated; it also requires us to write extra code to handle the various permuations of problems that can happen.

What we would prefere is if there was a piece of server software that essentially replicates Amazon S3's API. That way our clients can install the server on a box; and we don't need to change any code. So ... is there any such software out there?

Answer 1

Minio will be useful for this. Written in Go and simple to deploy.

Binary downloads here: https://min.io/download

To run as as an S3 server:

minio server path/to/disk/storage

Access key and secret will be output to the console.

Answer 2

This is possible via OpenStack Object Storage (code-named Swift), which is open source software for creating redundant, scalable object storage using clusters of standardized servers, specifically its recently added (optional) S3 API layer, which emulates the S3 REST API on top of Object Storage.

See Configuring Object Storage with the S3 API for the official documentation - a more insightful and illustrated small tutorial regarding the entire setup is available in S3 APIs on OpenStack Swift (which builds on the more complex Installing an OpenStack Swift cluster on EC2 though).

---

Update

An noteworthy alternative is Ceph, which is a unified, distributed storage system designed for excellent performance, reliability and scalability - interestingly it provides all three common storage models, i.e. Object Storage, Block Storage and a File System and the RADOS Gateway provides Amazon S3 and OpenStack Swift compatible interfaces to the RADOS object store [emphasis mine], see RADOS S3 API for details on currently supported S3 API features.

Answer 3

Have you looked at Cloudian? We use it internally at our company to develop our S3 app. I'm using the Community Edition which is free for up to 10TB of storage. It's got pretty good S3 coverage or at least covers most of the stuff my app uses (I use versioning and multipart uploads so I think my app is advanced). The version-ids and multipart ids etc that it generates are different than those you get from AWS but boto has no complaints so far. It also works with s3fs and other s3 bucket browsers that I have tried.

In my opinion it's a good tool for development against the AWS S3 API and should meet your requirements. You can point your app at your local Cloudian server and then when you are ready for production you can point it back at Amazon. Your mileage may vary... Good luck.

Answer 4

I recently started using Skylable for my S3 needs, it's free (GPL). Their object storage supports replication, HA and deduplication and it's fully S3 compatible. You can run their software on a single server (iron, virtual machine or container) if you don't need redundancy or you can use more nodes if you need HA.

The number of replicas can be chosen per bucket, just like with Swift. I started with 2 nodes in replica 2 and added more nodes as our userbase started growing, to cope with the extra network traffic and the space requirements.

Adding more nodes is really easy and can be done on a live cluster.

In my experience Skylable proved to be faster and more reliable than Swift. It's written in C and OCaml, it's not interpreted. The memory footprint is really low, so I can run a node even on some cheap VPS.

Recently they announced to be working on Swift APIs, apparently their goal is to replace Swift.

Answer 5

We ran into the problem of testing our S3 based code locally and actually implemented a small Java server, which emulates the S3 object API. As it might be useful to others, we setup a github repo along with a small website: http://s3ninja.net - all OpenSource under the MIT license.

Being quite simple and minimalistic, this tool is perfect for testing and developement purposes. However, to use in in production, one might want to add some security (altough the AWS hashes are already verified in the API - just the GUI is completely unprotected). Also, it doesn't do any replication or scaling. So this wouldn't be a good choice for large setups.

Answer 6

As it was already mentioned: you could try to use Swift as Amazon S3 alternative. Take a look at SwiftFS filesystem, it let you mount OpenStack container stored in Swift as a local filesystem.

Answer 7

While the original question is about S3 compatible software for (what it seems) production use, many are interested in the same software, but for local development and testing.

Regarding production-ready solution, there's a great S3 compatible storage software called Riak CS, it's solid and production proven for years, one drawback – it's not simple to setup. There are some limitations, but nothing major that gets in the way, see api / compatibility documentation.

Already a few great answers for development and testing, this will be useful for Docker users – there's docker-riak-cs image that allows to quickly launch Riak CS instance. I've been using it for nearly 2 years for local development and integration testing with great success.

Answer 8

If you want to have an S3-like API but hosting the data yourself the mention of minio is spot on.

If you want to write services that interact with S3 but you want to test them locally for speed you can use Localstack which I don't think I've seen mentioned here. It emulates not only S3 but many other AWS services as well. I wouldn't recommend using it for actual customer files as it is an in-memory datastore for testinng only.

https://github.com/localstack/localstack

https://bluesock.org/~willkg/blog/dev/using_localstack_for_s3.html