IAM Role Attached to Instance "Unable to Locate Credentials" - Can't hit metadata endpoint

Question

I am working with an EC2 instance with a custom Ubuntu 16.04 AMI.

I'm attempting to migrate to using an IAM role attached to the instance for accessing S3.

I have created a simple test role with the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

and the following policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This was attached to an already running EC2 instance.

On the EC2 instance, I tested out the CLI in a couple of different ways, and I'm getting surprising results.

ubuntu@machine:~$ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".

ubuntu@machine:~$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key                <not set>             None    None
secret_key                <not set>             None    None
    region                <not set>             None    None

I then went to check the metadata endpoint to confirm the role is attached, but ran into some issues. My suspicion is this is the root of the problem, but I've never seen this before - an inability to access the metadata endpoint. Does anyone know what might be blocking this or why this is unavailable?

ubuntu@machine:~$ curl http://169.254.169.254/latest/meta-data/
curl: (7) Couldn't connect to server

Many Thanks

Answer 1

Just to close the loop on this and help out other people that may encounter this issue.

This seemed to be an IP route that was blocked by the package cloud-init

https://github.com/cloud-init/cloud-init/blob/master/cloudinit/config/cc_disable_ec2_metadata.py#L36.

Running sudo ip route del prohibit 169.254.169.254 allowed access to the metadata endpoint, and the attached IAM role became usable.