Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can't run binary from within python aws lambda function

I am trying to run this tool within a lambda function: https://github.com/nicolas-f/7DTD-leaflet

The tool depends on Pillow which depends on imaging libraries not available in the AWS lambda container. To try and get round this I've ran pyinstaller to create a binary that I can hopefully execute. This file is named map_reader and sits at the top level of the lambda zip package.

Below is the code I am using to try and run the tool:

command = 'chmod 755 map_reader'
args = shlex.split(command)
print subprocess.Popen(args)

command = './map_reader -g "{}" -t "{}"'.format('/tmp/mapFiles', '/tmp/tiles')
args = shlex.split(command)
print subprocess.Popen(args)

And here is the error, which occurs on the second subprocess.Popen call:

<subprocess.Popen object at 0x7f08fa100d10>
[Errno 13] Permission denied: OSError

How can I run this correctly?

like image 580
stevepkr84 Avatar asked Jan 14 '17 14:01

stevepkr84


People also ask

Can Lambda run binaries?

The Lambda execution environment is based on a specific Amazon Linux AMI and kernel version. Any native binaries that are used in a Lambda deployment package must be compiled in this environment, and only 64-bit binaries are supported.

What are the major limitations of Lambda function?

Technical LimitationsThe maximum time a function can run is 15 minutes, and the default timeout is 3 seconds. Obviously, this makes Lambda unsuitable for long-running workloads. The payload for each invocation of a Lambda function is limited to 6MB, and memory is limited to just under 3GB.

Which programming language is currently not supported by Lambda?

AWS Lambda natively supports Java, Go, PowerShell, Node. js, C#, Python, and Ruby code, and provides a Runtime API which allows you to use any additional programming languages to author your functions. Please read our documentation on using Node.


2 Answers

You may have been misled into what the issue actually is.

I don't think that the first Popen ran successfully. I think that it just dumped a message in standard error and you're not seeing it. It's probably saying that

chmod: map_reader: No such file or directory

I suggest you can try either of these 2:

  1. Extract the map_reader from the package into /tmp. Then reference it with /tmp/map_reader.
  2. Do it as recommended by Tim Wagner, General Manager of AWS Lambda who said the following in the article Running Arbitrary Executables in AWS Lambda:

Including your own executables is easy; just package them in the ZIP file you upload, and then reference them (including the relative path within the ZIP file you created) when you call them from Node.js or from other processes that you’ve previously started. Ensure that you include the following at the start of your function code:

process.env[‘PATH’] = process.env[‘PATH’] + ‘:’ + process.env[‘LAMBDA_TASK_ROOT’]

The above code is for Node JS but for Python, it's like the following

import os os.environ['PATH']

That should make the command command = './map_reader <arguments> work.

If they still don't work, you may also consider running chmod 755 map_reader before creating the package and uploading it (as suggested in this other question).

like image 67
Jeshan Avatar answered Sep 21 '22 23:09

Jeshan


Like the docs mention for Node.js, you need to update the $PATH, else you'll get command not found when trying to run the executables you added at the root of your Lambda package. In Node.js, that's:

process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT']

Now, the same thing in Python:

import os

# Make the path stored in $LAMBDA_TASK_ROOT available to $PATH, so that we
# can run the executables we added at the root of our package.
os.environ["PATH"] += os.pathsep + os.environ['LAMBDA_TASK_ROOT']

Tested OK with Python 3.8.

(As a bonus, here are some more env variables used by Lambda.)

like image 36
Fabien Snauwaert Avatar answered Sep 21 '22 23:09

Fabien Snauwaert