How to grep into files stored in S3

Question

Do anybody know how to perform grep on S3 files with aws S3 directly into the bucket? For example I have FILE1.csv, FILE2.csv with many rows and want to look for the rows that contain string JZZ

aws s3 ls --recursive s3://mybucket/loaded/*.csv.gz | grep ‘JZZ’

Answer 1

The aws s3 cp command can send output to stdout:

aws s3 cp s3://mybucket/foo.csv - | grep 'JZZ'

The dash (-) signals the command to send output to stdout.

See: How to use AWS S3 CLI to dump files to stdout in BASH?

Answer 2

You can do it locally with the following command:

aws s3 ls --recursive s3://<bucket_name>/<path>/ | awk '{print $4}' | xargs -I FNAME sh -c "echo FNAME; aws s3 cp s3://<bucket_name>/FNAME - | grep --color=always '<regex_pattern>'"

Explanation: The ls command generates a list of files then we select the file name from the output and for each file (xargs command) download the file from S3 and grep the output.

I don't recommend this approach if you have to download a lot of data from S3 (due to transfer costs). You can avoid the costs for internet transfer though if you run the command on some EC2 instance that is located in a VPC with an S3 VPC endpoint attached to it.