Something like OverlayFS for s3?

Question

Is there something like OverlayFS, but for aws s3?

Our production system stores data in aws s3.

We have several test systems. Up to now we copy all data from the production s3 to s3 of the test system.

This is slow.

It would be great if there would be something like OverlayFS, but for s3:

• Writes should go the test system
• Reads should first check if there was a write to the test system. If yes, take this.
• else (not write was done yet), then the read should go to the production system.

At the end I want a s3 API (not a file-system)

Answer 1

I had this issue and ended up creating a fork of S3Proxy with support for an "OverlayBlobstore".

I wouldn't say that it's production-ready, and I'm not sure I'm motivated enough to ever bring it to that point but it works well enough for testing purposes.

$ docker run --rm -it switchmedia/s3proxy:latest \
  --publish 80:80
  --env S3PROXY_AUTHORIZATION=none \
  --env JCLOUDS_IDENTITY=$S3_ACCESS_KEY \
  --env JCLOUDS_CREDENTIAL=$S3_SECRET_KEY \
  --env JCLOUDS_PROVIDER=s3 \
  --env S3PROXY_OVERLAY_BLOBSTORE=true

$ curl -XGET localhost/bucket_name

Note: I strongly suggest you use credentials that are limited to Read Only. That way if there is a bug in the software, or the overlay is disabled for some reason, writes don't leak through.

Docker Image GitHub Repo