Elastic Beanstalk App in Private VPC without NAT

Question

Is it possible to deploy an EB app in a custom private VPC, without using a NAT Gateway? I want to use AWS API Gateway as the public entry point for the app while keeping the services private. After setting everything up, my environment keeps failing with:

The EC2 instances failed to communicate with AWS Elastic Beanstalk, either because of configuration problems with the VPC or a failed EC2 instance. Check your VPC configuration and try launching the environment again.

The VPC consists of:

• Two private subnets for the EC2 instances
• Two private subnets for the load balancer

The template I used is vpc-private.yaml from the EB samples repo.

The EB app is a multi-container docker application with a Nginx reverse proxy and a Flask app running with uWSGI (similar to this configuration). I'm using an internal Network Load Balancer and the instances are not assigned public IP addresses. The Docker images are hosted in AWS ECR.

A common answer in forums seems to be to have one public subnet with NAT to allow internet access. AWS documentation indicates that using a private/private configuration is possible though by using VPC Endpoints, as long as the app does not require an internet connection.

The endpoints I have configured are:

com.amazonaws.us-east-1.s3
com.amazonaws.us-east-1.elasticbeanstalk
com.amazonaws.us-east-1.ecs-telemetry
com.amazonaws.us-east-1.ecs-agent
com.amazonaws.us-east-1.ecs
com.amazonaws.us-east-1.ecr.dkr
com.amazonaws.us-east-1.ecr.api
com.amazonaws.us-east-1.elasticbeanstalk-health

It looks like VPC endpoints for Beanstalk are fairly new and I could not find any other resource explaining how to do this...

Thanks!

EDIT (solved)

A few setup details I forgot to mention:

• DNS resolution and DNS hostnames are enabled in the VPC
• Private DNS is enabled for every endpoint
• Made sure that the endpoint's security group allowed the instance's VPC as inbound traffic

Thanks to @Marcin suggestion, I was able to log into the instances via Session Manager (just add ssm, ec2messages and ssmmessages endpoints). After checking the logs, I noticed the following error:

2020-08-01 15:15:56,801 [WARNING] Timeout of 60 seconds breached
2020-08-01 15:15:56,801 [ERROR] Client-side timeout
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 162, in _retry
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 231, in _timeout
    raise TimeoutError("Execution did not succeed after %s seconds" % duration)
TimeoutError

I found other people had the same issue, and it was caused by cfn-signal failing to execute. I was able to solve this by adding the cloudformation endpoint. On the following deployment I now noticed a new sqs connection error which was solved by adding the sqs endpoint.

After adding the mentioned endpoints to the VPC, the deployment succeeded. Either way, I'm not sure why these endpoints were necessary, and why it was not mentioned in the AWS documentation in the first place.

Answer 1

The template you linked does not have NAT, even though the description says it has. AWS mistake.

The template creates two private subnets, fully cut off from the internet and any other services, such as EB or ECR.

The important thing to note, and the template fails to showcase it, is to enable private DNS for your endpoints.

For that to work you must ensure that EnableDnsHostnames and EnableDnsSupport are enabled for the VPC. The template from the docs does not enable them.

These could possibly explain why your VPC interface endpoints don't work.