Is it possible to write to s3 via a stream using s3 java sdk

Question

Normally when a file has to be uploaded to s3, it has to first be written to disk, before using something like the TransferManager api to upload to the cloud. This cause data loss if the upload does not finish on time(application goes down and restarts on a different server, etc). So I was wondering if it's possible to write to a stream directly across the network with the required cloud location as the sink.

Answer 1

It is possible:

AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
    .build();
s3Client.putObject("bucket", "key", youtINputStream, s3MetData)

AmazonS3.putObject

Answer 2

You don't say what language you're using, but I'll assume Java based on your capitalization. In which case the answer is yes: TransferManager has an upload() method that takes a PutObjectRequest, and you can construct that object around a stream.

However, there are two important caveats. The first is in the documentation for PutObjectRequest:

When uploading directly from an input stream, content length must be specified before data can be uploaded to Amazon S3

So you have to know how much data you're uploading before you start. If you're receiving an upload from the web and have a Content-Length header, then you can get the size from it. If you're just reading a stream of data that's arbitrarily long, then you have to write it to a file first (or the SDK will).

The second caveat is that this really doesn't prevent data loss: your program can still crash in the middle of reading data. One thing that it will prevent is returning a success code to the user before storing the data in S3, but you could do that anyway with a file.