Lambda in VPC won't create new ENI after an ENI has been manually detached from subnet

Question

I have run into a problem where my Lambda functions can no longer create new ENI's in a VPC subnet once an ENI has been detached and deleted from that subnet. I have reproduced this on two different VPC's and multiple subnets.

Here are the steps I used to reproduce this:

1. Create a Lambda function and associate it with a VPC and a subnet.
2. Execute the Lambda function which will cause it to create a new ENI in the subnet. The function is able to access an RDS database in the VPC.
3. Manually detach and delete the ENI from the subnet.
4. Execute the Lambda function again. No ENI will be created. Lambda function is unable to access RDS database.

I repeated these steps across multiple subnets in two VPC's and a now Lambda functions are unable to create new ENI's in any of those subnets.

It seemed like the VPC subnet was permanently unable to have new ENI's created by Lambda functions but after letting things sit overnight I ran the function again this morning and it was able to create an ENI. AWS must have an automated process that cleaned something up. Even so, I tested detaching and deleting again and I'm back in the state where the Lambda function won't create a new ENI.

Not sure what is going on here. I'm not selecting 'Force Detach' when detaching the ENI. What am I doing wrong?

Answer 1

From the research I have done this afternoon it appears this is a limitation of Lambda functions running in a VPC and has been for over a year. There is a six hour delay in the clean up of ENI's after a Lambda function has been deleted. My manual detachment and deletion of the ENI made things worse because it seems the ENI isn't really removed until after the six hour delay. The manual deletion seems to put the ENI in some sort of orphaned state that prevents new ENI's from being created in that subnet until after AWS has cleaned up the old one.

https://www.reddit.com/r/aws/comments/4fncrl/dangling_enis_after_deleting_an_invpc_lambda_with/

https://github.com/hashicorp/terraform/issues/5767

So it seems currently the only solution to forcing clean up is to delete the ENI and the subnet it was in. :-/

I let my serverless remove command run for a long time (without me speeding things along by manually deleting the ENI) and it did eventually finish but it took 40 minutes. The ENI was left behind but everything else was cleaned up. I imagine in six hours that ENI will disappear.

Answer 2

I recently faced a similar issue. I couldn't delete sg or subnets since they were being used by other stacks. I figured out if I force lambda to cold start, it might create new eni. To test this I allocated max memory and timeout. It worked.