How to integrate API Gateway with internal ALB

Question

According to https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-private.html it is possible to integrate API Gateway with an internal Application Load Balancer using a private VPC link.

However I cannot make it work.

I have a service accessible internally through the ALB. The ALB has no public IP, it balances requests in a AWS Fargate cluster (all within private subnets).

ubuntu@ip-10-0-40-89:~$ curl http://internal-sasw-alb-355535611.eu-west-1.elb.amazonaws.com/health
Assembly=Sasw.SampleService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null, Environment=Production, CurrentTime=05/05/2021 08:36:00

so all is good internally and I've got my service available for GET http://alb_url/health

I want to access the same service, only this time through an API Gateway. So I do the following:

1. I create a VPC link foo of HTTP API type associated to the VPC and, specifically, to the 3 private subnets where the EC2 instances (Fargate cluster) are 10.0.40.0/24, 10.0.50.0/24, 10.0.60.0/24
2. I create an API Gateway of type HTTP API foo.
3. I add an integration HTTP for method ANY and URL endpoint http://internal-rubiko-alb-355535611.eu-west-1.elb.amazonaws.com (I have also tried with http://internal-rubiko-alb-355535611.eu-west-1.elb.amazonaws.com/{proxy} and with http://internal-rubiko-alb-355535611.eu-west-1.elb.amazonaws.com/{proxy+} without success)
4. I leave the route as suggested Method ANY, resource path /%7Bproxy%7D and integration target ANY http://internal-rubiko-alb-355535611.eu-west-1.elb.amazonaws.com/{proxy}
5. I leave the stage name $default and Auto-deploy
6. Create. It gives me the invoke url https://7ki6gvrngl.execute-api.eu-west-1.amazonaws.com

A GET on https://7ki6gvrngl.execute-api.eu-west-1.amazonaws.com gets me a 404. A GET on https://7ki6gvrngl.execute-api.eu-west-1.amazonaws.com/health gets me a 404 A GET on https://7ki6gvrngl.execute-api.eu-west-1.amazonaws.com/$default/health gets me a 404

What am I doing wrong? Is it a routing issue? My security group accepts any protocol from anywhere. So it does my NACL. My routing table for the public subnets is

Destination   Target                Status   Propagated
10.0.0.0/16   local                 active   No 
0.0.0.0/0     igw-08d3973e93dd580cc active   No

For the private ones

Destination  Target                 Status   Propagated
10.0.0.0/16  local                  active   No 
0.0.0.0/0    nat-040aa996d0fb1a623  active   No

Thanks for your help.

---

UPDATE 1 (2021-05-05) I've been setting flow log on the VPC, plenty of traces for all network interfaces involved but I can't find anything meaningful there. They look like this:

2 754027052283 eni-06b3871889a039d54 10.0.50.60 10.0.30.69 47472 80 6 5 399 1620233998 1620233999 ACCEPT OK
2 754027052283 eni-06b3871889a039d54 10.0.30.69 10.0.50.60 80 47472 6 5 582 1620233998 1620233999 ACCEPT OK
2 754027052283 eni-06b3871889a039d54 172.245.79.11 10.0.30.69 25610 23 6 1 40 1620233998 1620233999 ACCEPT OK
2 754027052283 eni-06b3871889a039d54 10.0.30.69 172.245.79.11 23 25610 6 1 40 1620233998 1620233999 ACCEPT OK
2 754027052283 eni-0ec2ff9b148264fc1 - - - - - - - 1620233999 1620234010 - NODATA
2 754027052283 eni-0c2d904cc281e7772 185.153.199.146 10.0.10.108 58226 10089 6 1 40 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 92.63.197.95 10.0.10.108 55424 8820 6 1 40 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 147.203.255.20 10.0.10.108 59414 161 17 1 71 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 107.174.25.146 10.0.10.108 1406 23 6 1 40 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 45.146.164.93 10.0.10.108 52329 3835 6 1 40 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 192.241.159.103 10.0.10.108 38868 8088 6 1 40 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 162.142.125.145 10.0.10.108 43260 49501 6 1 44 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 104.206.128.38 10.0.10.108 57766 5985 6 1 44 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 198.98.54.148 10.0.10.108 54300 8443 6 1 44 1620234001 1620234061 REJECT OK
2 754027052283 eni-0c2d904cc281e7772 162.142.125.146 10.0.10.108 33318 15006 6 1 44 1620234001 1620234061 REJECT OK
2 754027052283 eni-02884e94ae95359e6 10.0.60.101 10.0.30.69 37440 80 6 5 399 1620234003 1620234005 ACCEPT OK
2 754027052283 eni-02884e94ae95359e6 10.0.30.69 10.0.60.101 80 37440 6 5 582 1620234003 1620234005 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.50.218 10.0.40.89 57566 1113 6 66 4862 1620234008 1620234127 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.40.89 10.0.60.197 1113 38912 6 122 7664 1620234008 1620234127 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.40.89 10.0.50.218 1113 57566 6 131 8230 1620234008 1620234127 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.60.197 10.0.40.89 38912 1113 6 62 4544 1620234008 1620234127 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.40.89 10.0.50.218 52812 2113 6 271 70540 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.40.89 10.0.60.197 2113 34170 6 161 41607 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.60.197 10.0.40.89 2113 42962 6 158 44927 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.40.89 10.0.50.218 2113 49902 6 165 43038 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.50.218 10.0.40.89 2113 52812 6 163 43127 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.60.197 10.0.40.89 34170 2113 6 263 68432 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.40.89 10.0.60.197 42962 2113 6 281 73877 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-05b8087c636058aad 10.0.50.218 10.0.40.89 49902 2113 6 270 70422 1620234008 1620234307 ACCEPT OK
2 754027052283 eni-06b3871889a039d54 10.0.30.69 10.0.50.60 80 47482 6 5 582 1620234010 1620234011 ACCEPT OK
2 754027052283 eni-06b3871889a039d54 10.0.50.60 10.0.30.69 47482 80 6 5 399 1620234010 1620234011 ACCEPT OK
2 754027052283 eni-0ae20fdd844224248 - - - - - - - 1620234012 1620234033 - NODATA
2 754027052283 eni-0ae20fdd844224248 - - - - - - - 1620234013 1620234026 - NODATA
2 754027052283 eni-030c3f3c47db36cb1 216.218.206.88 10.0.10.175 39712 30005 6 1 40 1620234014 1620234015 ACCEPT OK
2 754027052283 eni-030c3f3c47db36cb1 193.107.216.163 10.0.10.175 5065 5060 17 1 451 1620234015 1620234016 ACCEPT OK
2 754027052283 eni-0795bf1958db33aa7 - - - - - - - 1620234015 1620234032 - NODATA
2 754027052283 eni-0ae20fdd844224248 - - - - - - - 1620234016 1620234033 - NODATA
2 754027052283 eni-0ec2ff9b148264fc1 - - - - - - - 1620234017 1620234031 - NODATA

I don't even know how to interpret those.

I tried generating the private link, API Gateway and private ALB with cloudformation. No luck either. I keep getting not found when accessing service urls through Api Gateway

AWSTemplateFormatVersion: 2010-09-09
Description: API Gateway
Parameters:
  prefix:
    Type: String
    Description: The prefix namespace or company name
    Default: sasw
Resources:
  apiGateway:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      Name: !Sub ${prefix}-api-gateway
      Description: Api Gateway for http
      ProtocolType: HTTP
  apiRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref apiGateway
      RouteKey: ANY /
      Target: !Join
        - /
        - - integrations
          - !Ref apiAlbIntegration
  privateLink:
    Type: AWS::ApiGatewayV2::VpcLink
    Properties:
      Name: !Sub ${prefix}-private-link
      SecurityGroupIds:
        - Fn::ImportValue: !Sub ${prefix}-web-sg-id
      SubnetIds:
        - Fn::ImportValue: !Sub ${prefix}-private-a-id
        - Fn::ImportValue: !Sub ${prefix}-private-b-id
        - Fn::ImportValue: !Sub ${prefix}-private-c-id
  apiAlbIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref apiGateway
      Description: Private ALB integration
      IntegrationType: HTTP_PROXY
      IntegrationMethod: ANY
      ConnectionType: VPC_LINK
      ConnectionId: !Ref privateLink
      IntegrationUri: 
        Fn::ImportValue: !Sub ${prefix}-alb-http-listener-id
      PayloadFormatVersion: 1.0
  apiStage:
    Type: AWS::ApiGatewayV2::Stage
    Properties:
      StageName: $default
      AutoDeploy: true
      ApiId: !Ref apiGateway
Outputs:
  apiUrl:
    Description: Invoke URL
    Value: !Sub https://${apiGateway}.execute-api.${AWS::Region}.amazonaws.com/

Running out of ideas.

---

UPDATE 2 (2021-05-06) I found gold in this article https://medium.com/swlh/aws-api-gateway-private-integration-with-http-api-and-a-vpc-link-602360a1cd84

I can make a sample work if I create a route /health with private integration to my private ALB. Once I find a way to have a wildcard route that forwards any url/verb to ALB I'll answer my own question.

Answer 1

I got it working. It is definitely possible to use API Gateway http integrated with a private (i.e: internal facing) ALB that balances traffic in private subnets.

The problem I had is that when I created the API in API Gateway through the console, there is an option to add integration, but that integration at that point only allows HTTP or Lambda, and I don't want that, I want a private integration using a VPC link I create in advance.

So here are the steps:

1. Create (if non existing already) a security group that allows HTTP traffic on 80. This group will be associated later on to VPC link
2. Create VPC link associated to the VPC and, explicitly, to the private subnets where the EC2services or fargate cluster are. Make sure you select the security group that allows HTTP traffic
3. Create HTTP API in Apu Gateway. On the first step give it a name but DO NOT create an integration just yet. Skip that. Skip the route creation also. Choose a stage name or leave the $default (I use $default and auto-deploy).
4. Create a route. If you want to accept anything, do so by choosing ANY and the path /{proxy+}.
5. Finally, on that route, attach an integration. This time you'll see that there is an option to choose a private resource where you can explicitly select the private ALB with its http listener AND the VPC link created previously.

That's it. Http requests to Api Gateway will be directed to the private internal facing ALB.