How to get Glue development endpoint URL inside CloudFormation template?

Question

I'm trying to write a CloudFormation template that creates:

• Glue development endpoint (type AWS::Glue::DevEndpoint); and
• EC2 instance with Zeppelin notebook connected to the created endpoint.

The second resource requires endpoint URL. AWS::Glue::DevEndpoint provides endpoint name in the output, but it's not clear how to fetch the URL.

The only solution I found so far is to introduce aws glue get-dev-endpoint command and parse its output at some point of instance initialization process.

Answer 1

As of May 2019, Cloudformation does not support retrieving the public address out of a AWS::Glue::DevEndpoint resource.

As you suggested in the question, you can get the dev endpoint name from CF and pass that to an AWS CLI command in the user data of an EC2 instance which is probably the best way to retrieve that.

I think the model error you are getting with the glue CLI may be resolved if you ensure you are using the latest version of the AWS CLI before you run the command. This error makes it sound like there is some sort of config issue with the glue service in the CLI.

Resources:
   MyDevEndpoint:
      Type: AWS::Glue::DevEndpoint
      Properties: 
         ...

   MyInstance:
      Type: AWS::EC2::Instance
      Properties:
         ...
         UserData:
            Fn::Base64: !Sub |
               aws --version
               yes | pip3 install awscli --upgrade --user
               aws --version
               ENDPOINT_NAME=${MyDevEndpoint}
               aws glue get-dev-endpoint --endpoint-name $ENDPOINT_NAME
               ...