Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS: how to fix S3 event replacing space with '+' sign in object key names in json

I have a lamba function to copy objects from bucket 'A' to bucket 'B', and everything was working fine, until and object with name 'New Text Document.txt' was created in bucket 'A', the json that gets built in S3 event, key as "key": "New+Text+Document.txt".

the spaces got replaced with '+'. I know it is a known issue by seraching on web. But I am not sure how to fix this and the incoming json itself has a '+' and '+' can be actually in the name of the file. like 'New+Text Document.txt'.

So I cannot blindly have logic to space '+' by ' ' in my lambda function.

Due to this issue, when code tries to find the file in bucket it fails to find it.

Please suggest.

like image 533
ViS Avatar asked Jun 27 '17 11:06

ViS


People also ask

How do I protect my S3 from overwriting?

With S3 Object Lock, you can store objects using a write-once-read-many (WORM) model. Object Lock can help prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely.

Does S3 support JSON?

Amazon S3 Select works on objects stored in CSV, JSON, or Apache Parquet format. It also works with objects that are compressed with GZIP or BZIP2 (for CSV and JSON objects only), and server-side encrypted objects.

Does S3 Putobject overwrite?

If an object already exists in a bucket, the new object will overwrite it because Amazon S3 stores the last write request.

Which of the object attributes uniquely identify the S3 object within a bucket?

The object key (or key name) uniquely identifies the object in an Amazon S3 bucket. Object metadata is a set of name-value pairs. For more information about object metadata, see Working with object metadata. When you create an object, you specify the key name, which uniquely identifies the object in the bucket.


1 Answers

I came across this looking for a solution for a lambda written in python instead of java; "urllib.parse.unquote_plus" worked for me, it properly handled a file with both spaces and + signs:

from urllib.parse import unquote_plus import boto3   bucket = 'testBucket1234' # uploaded file with name 'foo + bar.txt' for test, s3 Put event passes following encoded object_key object_key = 'foo %2B bar.txt' print(object_key) object_key = unquote_plus(object_key) print(object_key)  client = boto3.client('s3') client.get_object(Bucket=bucket, Key=object_key) 
like image 170
kinzleb Avatar answered Oct 11 '22 09:10

kinzleb