s3 proxy on kubernetes using Ingress

Question

I am using this ingress controller and would like to setup a s3 proxy to some bucket. If I call in a browser the url

https://my-kube-server.org/img/dog.jpg 

I expect to see/download the image at

https://s3.eu-central-1.amazonaws.com/mybucket123/pictures/dog.jpg

I can setup a rewrite rule and point to an external service as explained in this example:

kind: Service
apiVersion: v1
metadata:
  name: s3-proxy
spec:
  type: ExternalName
  externalName: s3.eu-central-1.amazonaws.com
  headers:
  - host: s3.eu-central-1.amazonaws.com

But I get errors from aws because it's required to have "Host:s3.eu-central-1.amazonaws.com" in the header. I cannot set this header neither in the s3-proxy service definition nor in the ingress rule (configuration-snippet doesn't work because it will add another Host header after it's set already in the nginx.conf pod.

My solution is to take the whole location block for this ingress rule and to include it as a server-snippet, which is pretty brute force.

Another option is to have an nginx pod+service behind ingress that takes care of setting the right headers. So the flow would be request -> ingress-controller -> nginx -> s3.

Has anybody an idea how to proxy s3?

Answer 1

One of the possible solutions is to start the pods on each cluster node using DaemonSet that connect the S3 storage to the local directory using s3fs.

S3FS-FUSE:This is a free, open-source FUSE plugin and an easy-to-use utility which supports major Linux distributions & MacOS. S3FS also takes care of caching files locally to improve performance. This plugin simply shows the Amazon S3 bucket as a drive on your system.

Here is a good article that gives you step-by-step instructions on how to do it.

• Kubernetes shared storage with S3 backend

Then you can use this directory as a Volume in your Pods, for example, as a directory with a static content for your proxy server.

{ Or you can create a custom proxy server image with the s3fs tool inside and mount your S3 bucket directly into the Pod. Check out this and this articles for the details.
UPD: (This solution doesn't work yet because of limited support of FUSE in Kubernetes - FUSE volumes #7890) There is a workaround that require to run a privileged container }

There are two alternatives to s3fs available:

• ObjectiveFS - commercial FUSE plugin which supports Amazon S3 and Google Cloud Storage backends
• RioFs - lightweight utility written using the C language. Doesn’t support appending to file, doesn’t support fully POSIX-compliant file system interface, and it can’t rename folders.

Alternatively, you could try Traefik ingress controller:

• traefik.ingress.kubernetes.io/redirect-regex: ^http://localhost/(.*) - Redirect to another URL for that frontend. Must be set with traefik.ingress.kubernetes.io/redirect-replacement.
• traefik.ingress.kubernetes.io/redirect-replacement: http://mydomain/$1 - Redirect to another URL for that frontend. Must be set with traefik.ingress.kubernetes.io/redirect-regex.

Answer 2

If you need to use the server-snippet solution. Something similar to the following will add a custom location block to the Nginx ingress config. This is especially useful if you have other custom requirements for the Nginx ingress.

metadata:
  annotations:
    <other annotations>
    nginx.ingress.kubernetes.io/server-snippet: |
      location "/img/" {
        proxy_pass http://s3.eu-central-1.amazonaws.com/mybucket123/pictures/;
        proxy_set_header Host s3.eu-central-1.amazonaws.com;
      }

Also, if you are reverse proxying to an endpoint that provides limited logging (AWS S3). The Nginx mirror feature can help debug what requests are being sent. Add an additional annotation to the Ingress controller and specify a system where you can monitor the requests.

nginx.ingress.kubernetes.io/mirror-target: "http://my-debug-target/"