Lambda in VPC deletion takes more time

Question

I have created a stack that lambda in VPC using cloud formation. When I try to delete the entire stack, it takes 40-45 minutes of time.

My Iam Role has the following permission:

Action:                            
          - ec2:DescribeInstances
          - ec2:CreateNetworkInterface
          - ec2:AttachNetworkInterface
          - ec2:DescribeNetworkInterfaces
          - ec2:DeleteNetworkInterface
          - ec2:DetachNetworkInterface
          - ec2:ModifyNetworkInterfaceAttribute
          - ec2:ResetNetworkInterfaceAttribute
          - autoscaling:CompleteLifecycleAction
          - iam:CreateRole
          - iam:CreatePolicy
          - iam:AttachRolePolicy
          - iam:PassRole
          - lambda:GetFunction
          - lambda:ListFunctions
          - lambda:CreateFunction
          - lambda:DeleteFunction
          - lambda:InvokeFunction
          - lambda:GetFunctionConfiguration
          - lambda:UpdateFunctionConfiguration
          - lambda:UpdateFunctionCode
          - lambda:CreateAlias
          - lambda:UpdateAlias
          - lambda:GetAlias
          - lambda:ListAliases
          - lambda:ListVersionsByFunction
          - logs:FilterLogEvents
          - cloudwatch:GetMetricStatistics

How to improve the deletion time of the stack?

Answer 1

When a Lambda function executes within your VPC, an Elastic Network Interface (ENI) is created in order to give it network access. You can think of an ENI as a virtual NIC. It has a MAC address and at least one private IP address, and is "plugged into" any resource that connects to the VPC network and has an IP address inside the VPC (EC2 instances, RDS instances, ELB, ALB, NLB, EFS, etc.).

While it does not appear to be explicitly documented, these interfaces as used by Lambda appear to be mapped 1:1 to container instances, each of which hosts one or more containers, depending on the size of each container's memory allocation. The algorithm Lambda uses for provisioning these machines is not documented, but there is a documented formula for approximating the number that Lambda will create:

You can use the following formula to approximately determine the ENI requirements.

Projected peak concurrent executions * (Memory in GB / 3GB)

https://docs.aws.amazon.com/lambda/latest/dg/vpc.htm

This formula suggests that you will see more ENIs if you have either high concurrency or large memory footprints, or fewer ENIs if neither of those conditions is true. (The reason for 3GB boundary seems to be based on the smallest instance Lambda appears to use, in the background, which is the m3.medium general purpose EC2 instance. You can't see these among your EC2 instances, and you are not billed for them.)

In any event, Lambda doesn't shut down containers or their host instances immediately after function execution because it might need them for reuse on subsequent invocations, and since containers (and their host instances) are not destroyed right away, neither are their associated ENIs. To do so would be inefficient. In any event, the delay is documented:

There is a delay between the time your Lambda function executes and ENI deletion.

http://docs.aws.amazon.com/lambda/latest/dg/vpc.html

This makes sense, when we consider that the Lambda infrastructure's priorities should be focused on making resources available as needed and keeping them available for quick access performance reasons -- so tearing things down again is a secondary consideration that the service attends to in the background.

In short, this delay is normal and expected.

Presumably, CloudFormation has used tags to identify these interfaces, since it isn't readily apparent how to otherwise distinguish among them.

ENIs are visible in the EC2 console's left hand navigation pane under Network Interfaces, so it's possible that you could delete these yourself and hasten the process... but note that this action, assuming the system allows it, needs to be undertaken with due caution -- because if you delete an ENI that is attached to a container instance that Lambda subsequently tries to use, Lambda will not know that the interface is missing and the function will time out or throw an error at least until Lambda decides to destroy the attached container instance.