Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS CLI for S3 Select

I have the following code, which is used to run a SQL query on a keyfile, located in a S3 bucket. This runs perfectly. My question is, I do not wish to have the output written over to an output file. Could I see the output on the screen (my preference #1)? If not, what about an ability to append to the output file, rather than over-write it (my preference #2). I am using the AWS-CLI binaries to run this query. If there is another way, I am happy to try (as long as it is within bash)

aws s3api select-object-content \
    --bucket "project2" \
    --key keyfile1 \
    --expression "SELECT * FROM s3object s where Lower(s._1) = '[email protected]'" \
    --expression-type 'SQL' \
    --input-serialization '{"CSV": {"FieldDelimiter": ":"}, "CompressionType": "GZIP"}' \
    --output-serialization '{"CSV": {"FieldDelimiter": ":"}}' "OutputFile"
like image 937
rogerwhite Avatar asked Aug 17 '20 06:08

rogerwhite


People also ask

How do I use Amazon S3 select?

Getting started. Since S3 Select runs directly on S3 with data stored in your S3 bucket, all you need to get started is an AWS account and an S3 bucket. Sign in to your existing AWS account, or create a new AWS account. Once you sign in, create a S3 bucket to be used for testing with S3 Select.

What is S3 select in AWS?

S3 Select is a new Amazon S3 capability designed to pull out only the data you need from an object, which can dramatically improve the performance and reduce the cost of applications that need to access data in S3.

How do I pull data from AWS S3?

In the Amazon S3 console, choose your S3 bucket, choose the file that you want to open or download, choose Actions, and then choose Open or Download. If you are downloading an object, specify where you want to save it. The procedure for saving the object depends on the browser and operating system that you are using.

Does S3 have CLI interface?

You can access the features of Amazon Simple Storage Service (Amazon S3) using the AWS Command Line Interface (AWS CLI).


1 Answers

Of course, you can use AWS CLI to do this since stdout is just a special file in linux.

aws s3api select-object-content \
--bucket "project2" \
--key keyfile1 \
--expression "SELECT * FROM s3object s where Lower(s._1) = '[email protected]'" \
--expression-type 'SQL' \
--input-serialization '{"CSV": {"FieldDelimiter": ":"}, "CompressionType": "GZIP"}' \
--output-serialization '{"CSV": {"FieldDelimiter": ":"}}' /dev/stdout

Note the /dev/stdout in the end.

like image 81
jellycsc Avatar answered Oct 19 '22 05:10

jellycsc