Ansible Cloudwatch rule reports failed invocations

Question

I have created an AWS lambda that works well when I test it and when I create a cron job manually through a cloudwatch rule.

It reports metrics as invocations (not failed) and also logs with details about the execution.

Then I decided to remove that manually created cloudwatch rule in order to create one with ansible.

  - name: Create lambda service.
    lambda:
      name: "{{ item.name }}"
      state: present
      zip_file: "{{ item.zip_file }}"
      runtime: 'python2.7'
      role: 'arn:aws:iam::12345678901:role/lambda_ecr_delete'
      handler: 'main.handler'
      region: 'eu-west-2'
      environment_variables: "{{ item.env_vars }}"
    with_items:
      - name: lamda_ecr_cleaner
        zip_file: assets/scripts/ecr-cleaner.zip
        env_vars:
          'DRYRUN': '0'
          'IMAGES_TO_KEEP': '20'
          'REGION': 'eu-west-2'
    register: new_lambda

  - name: Schedule a cloudwatch event.
    cloudwatchevent_rule:
      name: ecr_delete
      schedule_expression: "rate(1 day)"
      description: Delete old images in ecr repo.
      targets:
        - id: ecr_delete
          arn: "{{ item.configuration.function_arn }}"
    with_items: "{{ new_lambda.results }}"

That creates almost the exact same cloudwatch rule. The only difference I can see with the manually created one is in the targets, the lambda version / alias is set to Default when created manually while it is set to version, with a corresponding version number when created with ansible.

The cloudwatch rule created with ansible has only failed invocations.

Any idea why this is? I can't see any logs. Is there a way I can set the version to Default as well with the cloudwatchevent_rule module in ansible?

Answer 1

I've lost hours with this too, same error and same confusion (Why there isn't a log for failed invokations?), I'm going to share my ""solution"", it will solve the problem to someone, and will help others to debug and find the ultimate solution.

Note: Be carefull, this could allow any AWS account execute your lambda functions

Since you got invoke the function by creating the rule target manually, I assume you added the invoke permission to the lambda from CloudWatch, however it looks like the Source Account ID is different when the event is created by cli/api and when is created by de AWS dashboard/console

If you are adding the Source Account condition in the lambda invoke permission from principal "events.amazonaws.com" to prevent any AWS account execute your lambdas just remove it (under your responsability!).

So, if your lambda policy looks like this:

{
    "Sid": "<sid>",
    "Effect": "Allow",
    "Principal": {
        "Service": "events.amazonaws.com"
    },
    "Action": "lambda:InvokeFunction",,
    "Condition": {
        "StringEquals": {
            "AWS:SourceAccount": "<account-id>"
        }
    },
    "Resource": "arn:aws:lambda:<region>:<account-id>:function:<lambda-function>"
}

Remove the "Condition" field

{
    "Sid": "sid",
    "Effect": "Allow",
    "Principal": {
        "Service": "events.amazonaws.com"
    },
    "Action": "lambda:InvokeFunction",,
    "Resource": "arn:aws:lambda:<region>:<account-id>:function:<lambda-function>"
}

And "maybe" it will work for you.

I think something weird it is happening with the cloudwatch event owner/creator data when the event is created by cli/api... maybe a bug? Not sure. I will keep working on it

Answer 2

To extend the answered here https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_Troubleshooting.html#LAMfunctionNotInvoked. Since you are creating it via API you should add permission to Lambda as mentioned before. Without compromising security you could do the following:

Add rule with PutRule api call, it will return you

{
   "RuleArn": "string"
}

Use the RuleArn in Lambda AddPermission call

aws lambda add-permission \
--function-name MyFunction \
--statement-id MyId \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn-from-PutRule-request