Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I implement a DynamoDB Global Secondary Index with Infrastructure As Code in CloudFormation

I am working on implementing a GSI in CloudFormation with Infrastructure As Code. All I want to do is use this table to keep count of the entries in the main DynamoTable. Here is what the main tale looks like:

Resources:
  CaseRecords:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.tableName}
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: caseRecordId
          AttributeType: S
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: caseRecordId
          KeyType: RANGE

I do not need the keys from the original table and all I want is to create a new HASH key for the new GSI that will tell me which table the count I am keeping track of came from, ie the table above.

Below is how I have tried to implement the GSI so far:

# Implement a GSI to handle item count totals
      GlobalSecondaryIndexes:
          - IndexName: gsiCaseCountTable
            KeySchema: 
              - AttributeName: table-name
                KeyType: HASH
            ProvisionedThroughput:
              ReadCapacityUnits: 5
              WriteCapacityUnits: 5

However, the error I get is as follows:

An error occurred: CaseRecords - Property Projection cannot be empty..

When I include the PROJECTION I was, with is only the userId from the original table simply to keep track of entry counts in the original table per user, I try the following:

 Implement a GSI to handle item count totals
      GlobalSecondaryIndexes:
      - IndexName: gsiCaseCountTable
        KeySchema:
        - AttributeName: table-name
          KeyType: HASH
        Projection:
          NonKeyAttributes:
          - userId
          ProjectionType: INCLUDE
        ProvisionedThroughput:
          ReadCapacityUnits: 5
          WriteCapacityUnits: 5

However this returns an error also:

An error occurred: CaseRecords - Property AttributeDefinitions is inconsistent with the KeySchema of the table and the secondary indexes.

How can I correctly implement a Global Secondary Index in Dynamo using a CloudFormation Template so I can record a count of entries in the original table????

Thanks.

UPDATE

In case anyone is wondering this is how I was able to deploy it. It is not a perfect solution but it lets me keep track and count the entries to the item table:

# NOTE: DynamoDB Serverless Configuration
# NoSQL Table for CaseRecord DB

Resources:
  CaseRecords:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.tableName}
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: caseRecordId
          AttributeType: S
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: caseRecordId
          KeyType: RANGE
      # Set the capacity based on the stage
      # ProvisionedThroughput:
        # ReadCapacityUnits: ${self:custom.tableThroughput}
        # WriteCapacityUnits: ${self:custom.tableThroughput}
      # Implement a GSI to handle item count totals
      GlobalSecondaryIndexes:
      - IndexName: gsiCaseCountTable
        KeySchema:
        - AttributeName: userId
          KeyType: HASH
        Projection:
          ProjectionType: KEYS_ONLY

UPDATE #2 - FAILING

Based on the info provided by @Pedro Arantes below, I am trying to implement the GSI with the Attribute Definitions that I want to use. This too however is failing. Below is the implementation and here is the link to the AWS Doc that I used: AWS GSI Doc and here is the failing implementation:

Resources:
  CaseRecords:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.tableName}
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: caseRecordId
          AttributeType: S
        - AttributeName: table-name
          AttributeType: S
        - AttributeName: count
          AttributeType: N
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: caseRecordId
          KeyType: RANGE
      # Set the capacity based on the stage
      # ProvisionedThroughput:
        # ReadCapacityUnits: ${self:custom.tableThroughput}
        # WriteCapacityUnits: ${self:custom.tableThroughput}
      # Implement a GSI to handle item count totals
      GlobalSecondaryIndexes:
      - IndexName: gsiCaseCountTable
        KeySchema:
        - AttributeName: table-name
          KeyType: HASH
        Projection: 
          NonKeyAttributes: 
            - userId
            - count
          ProjectionType: INCLUDE

How can I get this to work with only the NonKeyAttributes that I have declared in the AttributeDefinitions???

like image 310
lopezdp Avatar asked Jun 14 '19 16:06

lopezdp


1 Answers

You need to add table-name at AttributeDefinitions property. From docs:

AttributeDefinitions

A list of attributes that describe the key schema for the table and indexes. Duplicates are allowed.

So even if you don't use some attribute in the original table, you must declare to be able to use in your GSIs.

UPDATE #2 - FAILING

You're using keys attributes userId and count that you defined at AttributeDefinitions as NonKeyAttributes (but they are keys attributes) at Projection. You don't need to add them because they're automatically projected. From docs:

AWS::DynamoDB::Table Projection

Represents attributes that are copied (projected) from the table into an index. These are in addition to the primary key attributes and index key attributes, which are automatically projected.

FINAL TEMPLATE

Resources:
  CaseRecords:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.tableName}
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: caseRecordId
          AttributeType: S
        - AttributeName: table-name
          AttributeType: S
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: caseRecordId
          KeyType: RANGE
      # Set the capacity based on the stage
      # ProvisionedThroughput:
      # ReadCapacityUnits: ${self:custom.tableThroughput}
      # WriteCapacityUnits: ${self:custom.tableThroughput}
      # Implement a GSI to handle item count totals
      GlobalSecondaryIndexes:
        - IndexName: gsiCaseCountTable
          KeySchema:
            - AttributeName: table-name
              KeyType: HASH
          Projection:
            NonKeyAttributes:
              - count
            ProjectionType: INCLUDE

Considerations:

  1. count should not be on AttributeDefinitions because you're not using it as key.

  2. You don't need to add userId at Projection because it'll be project automatically since it's defined in AttributeDefinitions.

like image 165
Pedro Arantes Avatar answered Sep 23 '22 03:09

Pedro Arantes