Streaming S3 object to VertX Http Server Response

Question

The title basically explains itself.

I have a REST endpoint with VertX. Upon hitting it, I have some logic which results in an AWS-S3 object.

My previous logic was not to upload to S3, but to save it locally. So, I can do this at the response routerCxt.response().sendFile(file_path...).

Now that the file is in S3, I have to download it locally before I could call the above code.

That is slow and inefficient. I would like to stream S3 object directly to the response object.

In Express, it's something like this. s3.getObject(params).createReadStream().pipe(res);.

I read a little bit, and saw that VertX has a class called Pump. But it is used by vertx.fileSystem() in the examples.

I am not sure how to plug the InputStream from S3'sgetObjectContent() to the vertx.fileSystem() to use Pump.

I am not even sure Pump is the correct way because I tried to use Pump to return a local file, and it didn't work.

    router.get("/api/test_download").handler(rc -> {
        rc.response().setChunked(true).endHandler(endHandlr -> rc.response().end());
        vertx.fileSystem().open("/Users/EmptyFiles/empty.json", new OpenOptions(), ares -> {
            AsyncFile file = ares.result();
            Pump pump = Pump.pump(file, rc.response());
            pump.start();
        });
    });

Is there any example for me to do that?

Thanks

Answer 1

It can be done if you use the Vert.x WebClient to communicate with S3 instead of the Amazon Java Client.

The WebClient can pipe the content to the HTTP server response:

webClient = WebClient.create(vertx, new WebClientOptions().setDefaultHost("s3-us-west-2.amazonaws.com"));

router.get("/api/test_download").handler(rc -> {
HttpServerResponse response = rc.response();
response.setChunked(true);
webClient.get("/my_bucket/test_download")
 .as(BodyCodec.pipe(response))
  .send(ar -> {
    if (ar.failed()) {
      rc.fail(ar.cause());
    } else {
      // Nothing to do the content has been sent to the client and response.end() called
    }
  });
});

The trick is to use the pipe body codec.