possible ways to store JSON in dynamodb

Question

I am working on an implementation where my system will be integrated with an external application. This external application generates long and different json for every request.

I want to store this json in dynamodb, what would be the best approach to do that? Currently, I have created a POJO class with below property. In this definition property, I set the API response json as string.

@DynamoDBAttribute(attributeName = "definition")
private String definition;

Then create object of POJO, and save it to dynamodb using this.dynamoDBMapper.save(obj);

Questions:

1 - Is this a good approach to store JSON in dynamodb as string?

2 - What if I want to search by any property of this json in dynamodb?

3 - Is there a better way to store this json so it can be searchable also?

Also, this json is always different so I can't map it to POJO class, that's why I am trying to store it as a string.

Below is my sample POJO object that I am storing in dynamo db.

@DynamoDBTable(tableName = "job")
public class Job {

@DynamoDBHashKey(attributeName = "inbound_job")
private Long inboundId;

@DynamoDBAttribute(attributeName = "outbound_job")
private Long outboundId;

@DynamoDBAttribute(attributeName = "definition")
private String definition;
}

Below is respective json stored in dynamodb.

{
    "inbound_job": {
        "N": "3138788"
    },
    "outbound_id": {
        "N": "909092"
    },
    "jobDefinition": {
        "S": "{\r\n\t\"_id\": \"5ae1d9848376948a370d6962\",\r\n\t\"index\": 4,\r\n\t\"guid\": \"32bb8da0-a84a-4b3c-8775-c20216fa04b7\",\r\n\t\"isActive\": true,\r\n\t\"balance\": \"$2,683.96\",\r\n\t\"picture\": \"http:\/\/placehold.it\/32x32\",\r\n\t\"age\": 38,\r\n\t\"eyeColor\": \"brown\",\r\n\t\"name\": \"Swanson Hughes\",\r\n\t\"gender\": \"male\",\r\n\t\"company\": \"BIOSPAN\",\r\n\t\"email\": \"[email protected]\",\r\n\t\"phone\": \"+1 (999) 559-2315\",\r\n\t\"address\": \"851 Lyme Avenue, Shepardsville, American Samoa, 8015\",\r\n\t\"about\": \"Sit officia culpa in est Lorem. Officia occaecat nostrud Lorem officia non sint enim excepteur. Laboris dolore consectetur velit occaecat ullamco non nisi.\\r\\n\",\r\n\t\"registered\": \"2015-12-12T11:22:53 +08:00\",\r\n\t\"latitude\": 47.732205,\r\n\t\"longitude\": -164.823761,\r\n\t\"tags\": [\r\n\t\t\"velit\",\r\n\t\t\"anim\",\r\n\t\t\"id\",\r\n\t\t\"tempor\",\r\n\t\t\"et\",\r\n\t\t\"eu\",\r\n\t\t\"amet\"\r\n\t],\r\n\t\"friends\": [{\r\n\t\t\t\"id\": 0,\r\n\t\t\t\"name\": \"Kaye Fields\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"id\": 1,\r\n\t\t\t\"name\": \"Knapp Reed\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"id\": 2,\r\n\t\t\t\"name\": \"Hodge Morse\"\r\n\t\t}\r\n\t],\r\n\t\"greeting\": \"Hello, Swanson Hughes! You have 3 unread messages.\",\r\n\t\"favoriteFruit\": \"strawberry\"\r\n}"
    }
}

I also want to search by any json property of "definition" attribute.

Let me know your thoughts.

Regards.

Answer 1

Yes this is a good way to store your arbitrary json data.

You can then use document path in your queries and scans to find and retrieve parts of the responses. You don't need to do anything special when you save the json (other than make sure the json is properly escaped).

Its worth noting that you can only index top level DynamoDB item attributes. This means any 'search' you do on your response content will be a scan, which will evaluate every response in the table.

If there are any (reliable/consistent) attributes you want to index, you will need to extract them from the json and make them an attribute on the POJO.

Answer 2

Instead of using dynamoDBMapper, you can use Table.putItem. Refer DynamoDB Java docs for updating any free json into dynamodb.

Highlights from the above link (tweaked for your use-case):

    AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
        .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:8000", "us-west-2"))
        .build();
    DynamoDB dynamoDB = new DynamoDB(client);
    Table table = dynamoDB.getTable("YourTableName");
    try {
      table.putItem(new Item().withPrimaryKey("hashKeyName", hashKeyValue).withJSON("definition", definition);
      System.out.println("PutItem succeeded: " );
    }
    catch (Exception e) {
      System.err.println(e.getMessage());
      break;
    }