aws-sdk Multipart Upload to s3 with node.js

Question

I am trying to upload files to a s3 bucket using the node js aws-sdk V3.

I know I am supposed to be using the commmands: CreateMultipartUploadCommand,UploadPartCommandand so forth. But I can't find any working example of a full multipart upload.

Can anyone share any code samples?

Thanks in advance

Answer 1

I put together a bit of code for this: https://gist.github.com/kbanman/0aa36ffe415cdc6c44293bc3ddb6448e

The idea is to upload a part to S3 whenever we receive a chunk of data in the stream, and then finalize the upload when the stream is finished.

Complicating things is S3's minimum part size of 5MB on all but the last part in the series. This means that we need to buffer data until we can form those 5MB chunks. I accomplished this using a transformer that adds back-pressure on the content stream between each chunk upload.

Parallelization is also made difficult by the fact that S3 insists on receiving the parts in order (despite asking for the parts to be numbered).