Use AWS lambda to upload video into S3 with download URL

Question

I want to upload video into S3 using AWS lambda function. This video is not available in my local computer. I have 'download URL'. I don't want to download it into my local computer and upload it into S3. I am looking for a solution to directly put this video file into S3 using lambda function. If I use buffer or streaming, I will consume lots of memory. Is there a better efficient solution for this?

I really appreciate your help.

Answer 1

I had the same question and developed the following quick solution that is not dependent on the /tmp disk limits. It uses download stream as a file-like object.

Features:

• No external python modules, using AWS Lambda Python 3.6 builtin boto3 and urllib3
• Has chunked reading built in, good for downloading large file
• Efficient connection and memory usage through urllib3 pool management

• Using configurable upload_fileobj builtin multipart and threaded upload

  import boto3
  import botocore.vendored.requests.packages.urllib3 as urllib3
  
  def lambda_handler(event, context):
  
      url='http://yourdownloadurl/file.tgz' # put your url here
      bucket = 'aws-s3-bucket' #your s3 bucket
      key = 'folder/filename' #your desired s3 path or filename
  
      s3=boto3.client('s3')
      http=urllib3.PoolManager()
      s3.upload_fileobj(http.request('GET', url,preload_content=False), bucket, key)
  

Answer 2

You could certainly write an AWS Lambda function that would:

• Download the file from the URL and store it in /tmp
• Upload to Amazon S3 using the AWS S3 SDK

It would be easiest to download the complete file rather than attempting to stream it in 'bits'. However, please note that there is a limit of 500MB of disk space available for storing data. If your download is larger than 500MB, you'll need to do some creative programming to download portions of it and then upload it as a multi-part upload.

As for how to download it, use whatever library you prefer for download a web file.