Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Write csv file and save it into S3 using AWS Lambda (python)

I'm trying to write a csv file into an S3 bucket using AWS Lambda, and for this I used the following code:

data=[[1,2,3],[23,56,98]]
with open("s3://my_bucket/my_file.csv", "w") as f:
   f.write(data)

And this raises the following error:

[Errno 2] No such file or directory: u's3://my_bucket/my_file.csv': IOError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 51, in lambda_handler
with open("s3://my_bucket/my_file.csv", "w") as f:
IOError: [Errno 2] No such file or directory: u's3://my_bucket/my_file.csv'

Can I have some help with this please ?

PS: I'm using python 2.7

Thanking you in advance

like image 205
kab Avatar asked Apr 06 '18 14:04

kab


People also ask

How do I convert a CSV file to an S3 bucket?

Navigate to All Settings > Raw Data Export > CSV Upload. Toggle the switch to ON. Select Amazon S3 Bucket from the dropdown menu. Enter your Access Key ID, Secret Access Key, and bucket name.

Can Lambda upload file to S3?

Overview of serverless uploading to S3 Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda function. This gets a signed URL from the S3 bucket. Directly upload the file from the application to the S3 bucket.

How do you append data to an existing csv file in AWS S3 using Python boto3?

s3 has no append functionality. You need to read the file from s3, append the data in your code, then upload the complete file to the same key in s3.


1 Answers

Better to answer later than never. There are four steps to get your data in S3:

  • Call the S3 bucket
  • Load the data into Lambda using the requests library (if you don't have it installed, you are gonna have to load it as a layer)
  • Write the data into the Lambda '/tmp' file
  • Upload the file into s3

Something like this:

import csv
import requests
#all other apropriate libs already be loaded in lambda

#properly call your s3 bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
key = 'yourfilename.txt'

#you would need to grab the file from somewhere. Use this incomplete line below to get started:
with requests.Session() as s:
    getfile = s.get('yourfilelocation')

#Only then you can write the data into the '/tmp' folder.
with open('/tmp/yourfilename.txt', 'w', newline='') as f:
    w = csv.writer(f)
    w.writerows(filelist)
#upload the data into s3
bucket.upload_file('/tmp/yourfilename.txt', key)

Hope it helps.

like image 125
Pucho Avatar answered Oct 12 '22 16:10

Pucho