Type annotation for boto3 resources like DynamoDB.Table

Question

The boto3 library provides several factory methods that returns resources. For example:

dynamo = (
    boto3
    .resource('dynamodb')
    .Table(os.environ['DYNAMODB_TABLE'])
)

I want to annotate those resources so I can get better type checking and completion, but the only type alike I could find was from boto3.dynamodb.table import TableResource.

When I add that annotation:

dynamo: TableResource = (
    boto3
    .resource('dynamodb')
    .Table(os.environ['DYNAMODB_TABLE'])
)

The only method offered by auto-completion is batch_writer(self, overwrite_by_pkeys), even though the docs lists several others.

Is this the wrong class to use as annotation? Inspecting that variable type in the terminal I could see that it was <class 'boto3.resources.factory.dynamodb.Table'>, but it doesn't seem to be possible to get that type statically.

Answer 1

The types and API methods don't exist statically. boto3 uses data driven architecture, an extremely dynamic design that uses data in JSON format (here is an example) to determine what API calls are possible. They do this to make it easy to update the library to include new API changes. I'm not sure but I think they might use the same strategy for SDKs in other languages, so changes to multiple SDKs can be made with little duplicated work.

Here's a quote from their blog:

Libraries must adapt to changes in users’ needs and also to changes in the platforms on which they run. As AWS’s growth accelerated over the years, the speed at which our APIs are updated has also gotten faster. This required us to devise a scalable method to quickly deliver support for multiple API updates every week, and this is why AWS API support in Boto3 is almost completely data-driven. Boto3 has ‘client’ classes that are driven by JSON-formatted API models that describe AWS APIs, so most new service features only require a simple model update. This allows us to deliver support for API changes very quickly, in consistent and reliable manner.

You can also get a glimpse of this happening by stepping into a method call such as resource.Table in a debugger.

Answer 2

In addition to Alex Hall answer. Forward references can be used for solving this problem.

dynamo: 'boto3.resources.factory.dynamodb.Table' = (
    boto3
    .resource('dynamodb')
    .Table(os.environ['DYNAMODB_TABLE']))

Answer 3

It is possible to type annotate code which uses DynamoDB with help from this library: https://github.com/vemel/mypy_boto3.

Installation:

pip install boto3-stubs[dynamodb]

Usage:

from mypy_boto3_dynamodb import ServiceResource

dynamodb: ServiceResource = boto3.resource(
    "dynamodb", region_name=region
)
# now type checker or IDE can infer type of `table`
# and find its methods
table = dynamodb.Table("example")

It is also possible to annotate many other boto3 services, see the library's GitHub page.