How to programmatically list all aws resources and tags [closed]

Question

I would like to audit compliance with a tagging a scheme in AWS so I want to programmatically retrieve all assets from an account and inspect their tags.

Is there a reasonable way to achieve this without iterating through the fragmented aws clients in the boto or Java APIs? If I count the boto3 clients alone there are about 40 of them, most of them with just slightly differently semantics. If I actually did write code that uses all of them I'd have to add another client every time AWS roles out a new service.

What I've looked at already:
boto3 - 40 some per service clients to iterate makes it not feasible.
AWS Java client - as as above
AWS cli - same as above
Compliance Monkey (Netflix) - only looks at Auto Scaling Groups
AWS Config - You have to query the resources by tags which will fail to trace the untagged.

What I'm looking at soon:
Netflix Edda

What I've done for now:
Exactly what I didn't want, iterating through the boto3 cloudformation, ec2, s3, and autoscaling clients. This is better than nothing, but there are clearly gaps in this approach.

Answer 1

Been battling to get the script @whorka shared above to work but I think there must have been an update to the way skew scans resources. Anyway, the script below seems to do the trick!

    #!/usr/bin/env python
    import skew
    from skew.arn import ARN
    arn = ARN()
    services=arn.service.choices()
    services.sort()
    #services=["route53", "iam"]
    print('Enumerating all resources in the following services: ' + ' '.join(services) + '\n')
    for service in services:
      #skipping global services because the API endpoint fails due to it being a global service. Bug that needs fixing.
      if service == "iam" or service == "route53":
        print(service)  
        print('Skipping global services')
        #uri = 'arn:aws:' + service + '::*:*/*'
        #arn = skew.scan(uri)
        #for i in arn:
        #  print(i.arn)
      else:
        print('******' + service + '******')
        uri = 'arn:aws:' + service + ':*:*:*/*'
        arn = skew.scan(uri)
        for i in arn:
          print(i.arn)

There is a bug with Global Services, like IAM and Route53 as the API endpoint doesn't contain the region which causes skew to barf

Answer 2

Unfortunately, the solution you want doesn't exist.

Keep in mind that AWS doesnt support tagging of all of their services; so what you want to do may not even be possible (if what you want is some generic way to query all services and leverage common tags).

There exist services such as RightScale or as you know, Boto, that provide an abstraction on top of AWS. However, these third party abstractions will always have a lag time before they support a new service, compared to the AWS CLI.

If what you truly want is some generic way to query all services, AND handle the ones that don't currently support tagging, suggest you do one or all of the following:

1. Leverage tagging in all the services that currently support tagging: ec2, rds
2. Where a service does not support tagging, leverage the name of the resource to add the equivalent of "tags" e.g. <component>-<environment>-<resource type>
3. Write a small program that wraps/calls all the indivdual aws clis for each of the services.
4. In the program, convert the tags and resources names of each of the aws cli commands into a canonical format, stored in a datastore e.g. dynamo, berkleydb
5. Write a small program that then does your "compliance check" against the data in this datastore

When a new service comes online, tag your resources and extend your program in step 3 to support the new service. Note, the majority of recent new AWS services have gone through an announcement period, then preview period that lasted 6 months or more, e.g. Elastic File System, etc. So generally, you will have time before you will need to extend your program to support new services.

Also, keep in mind that of the "40 services" you mentioned, only a small subset can actually be "launched or tagged". Most of these services aren't truly "resources", e.g. AWS Config, CloudWatch, CloudTrail, SES, SNS, Data Pipeline. So what may feel like a lot of APIs to have to execute may end up being 10 or less.