Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

aws batch: submit job using lambda

Context: AWS, S3, Lambda, Batch.

I have a lambda that is triggered when a file is uploaded in a S3 Bucket. I want that the lambda submit a Batch job.

(edit: Between S3 and Lambda everything works fine. The problem is between Lambda and Batch.)

Q: What is the role I have to give to the lambda in order to be able to submit the batch job?

My lambda gets an AccessDeniedException and fail to submit the job when:

const params = {
  jobDefinition: BATCH_JOB_DEFINITION,
  jobName: BATCH_JOB_NAME,
  jobQueue: BATCH_JOB_QUEUE,
};

Batch.submitJob(params).promise() .then .......
like image 456
Costin Avatar asked Nov 16 '17 13:11

Costin


People also ask

Can we use AWS Lambda for batch processing?

Lambda function calls the Main batch orchestrator workflow to start the processing of the file. Main batch orchestrator workflow reads the input file and splits it into multiple chunks and stores them in an S3 bucket. Main batch orchestrator then invokes the Chunk Processor workflow for each split file chunk.

How do I submit a batch job to AWS?

To submit a jobOpen the AWS Batch console at https://console.aws.amazon.com/batch/ . From the navigation bar, select the AWS Region to use. In the navigation pane, choose Jobs, Submit job.

Can we call glue job from Lambda?

You can also trigger one or more Glue jobs from an external source such as an AWS Lambda function.


1 Answers

It seems that this was the role I was looking for: batch:SubmitJob. Using this role, the lambda was able to submit the job.

iamRoleStatements:
  - Effect: Allow
    Action:
      - batch:SubmitJob
    Resource: "arn:aws:batch:*:*:*"
like image 54
Costin Avatar answered Oct 03 '22 07:10

Costin