Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Upload uint8_t* buffer to AWS S3 without going via filesystem

Disclaimers: I am not a c++ programmer, please save me from myself.

I'm trying to create a PutObjectRequest in c++ using the AWS SDK.

I have a 'uint8_t*' (in Java-land from whence I hail we call this a byte[], I believe on planet c++ this is a buffer), and I need to get it into an Aws::IOStream somehow.

All of the examples show the data coming directly from the filesystem.

I've seen a couple of similar-ish (but not really) questions with answers that point to another third party library called Boost, but surely this is a common usecase? Why would I need a third party library to do something that should just be possible using the AWS SDK?:

"I have data, I want to put it up on S3. No it's not in the filesystem, yes I created it in memory."

uint8_t* buf; //<-- How do I get this...
...
Aws::S3::Model::PutObjectRequest object_request;
object_request.WithBucket(output_bucket).WithKey(key_name);

object_request.SetBody(data); //<-- ...into here

I really appreciate any help or pointers (no pun intended) here.

Update
I've tried everything in the comments, and this:

std::shared_ptr<Aws::IOStream> objectStream = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream");        
*objectStream << data;
objectStream->flush();
object_request.SetBody(objectStream);

and this:

std::shared_ptr<Aws::IOStream> objectStream = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream");        
std::istringstream is((char*) data);
*objectStream << is.rdbuf();
objectStream->flush();
object_request.SetBody(objectStream);

which compile, but each only uploads 2 bytes of data.

Other thing I've tried that don't compile are:

auto input_data = Aws::MakeShared<Aws::IOStream>("PutObjectInputStream", std::istringstream((char*) data), std::ios_base::in | std::ios_base::binary);
object_request.SetBody(input_data);

and

object_request.SetBody(std::make_shared<std::istringstream>( std::istringstream( (char*) spn ) ));

and these ones creates the object on S3, but with 0 bytes:

    std::shared_ptr<Aws::IOStream> objectStream = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream");
    objectStream->rdbuf()->pubsetbuf(static_cast<char*>(reinterpret_cast<char*>(data)), length);
    objectStream->rdbuf()->pubseekpos(length);
    objectStream->seekg(0);
    object_request.SetBody(objectStream);


    std::shared_ptr<Aws::IOStream> objectStream = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream");
    objectStream->rdbuf()->pubsetbuf(reinterpret_cast<char*>(data), length);
    objectStream->rdbuf()->pubseekpos(length);
    objectStream->seekg(0);
    object_request.SetBody(objectStream);
like image 913
ndtreviv Avatar asked Feb 07 '18 14:02

ndtreviv


People also ask

How do I upload directly to S3 bucket?

To upload folders and files to an S3 bucketSign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/ . In the Buckets list, choose the name of the bucket that you want to upload your folders or files to. Choose Upload.

What is the minimum file size that you can upload into S3?

Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 TB. The largest object that can be uploaded in a single PUT is 5 GB. For objects larger than 100 MB, customers should consider using the Multipart Upload capability.

What is the recommended way to upload a single large file to an S3 bucket?

Instead of using the Amazon S3 console, try uploading the file using the AWS Command Line Interface (AWS CLI) or an AWS SDK. Note: If you use the Amazon S3 console, the maximum file size for uploads is 160 GB. To upload a file that is larger than 160 GB, use the AWS CLI, AWS SDK, or Amazon S3 REST API.

How many ways you can upload data to S3?

There are three ways in which you can upload a file to amazon S3.


1 Answers

And after hours of hacking, here is the answer:

    Aws::S3::Model::PutObjectRequest object_request;
    object_request.WithBucket(output_bucket).WithKey(key_name);
    auto data = Aws::MakeShared<Aws::StringStream>("PutObjectInputStream", std::stringstream::in | std::stringstream::out | std::stringstream::binary);
    data->write(reinterpret_cast<char*>(buffer), length);
    object_request.SetBody(data);

Thanks to Ben Voigt for pointing out that when you make_shared (or MakeShared in my case), you're not actually passing it the data at that point. You're just telling it what T you're making shared.

Also helped by nbubis' answer to this question: const char * to std::basic_iostream

like image 53
ndtreviv Avatar answered Oct 06 '22 01:10

ndtreviv