Efficient substring Search in DynamoDB

Question

This is the context of my situation:

• I have a huge DB in dynamoDB with 250.000 items. (Example) table
• I want to be able to "substring search" through 3 attributes, getting the list of all items that match the substrings.
• The attributes i want to be able to search can have the same value among different items.
• My hash key is an id (the only attribute that really differentiates the items).
• I'm using react native as a client
• My schema has these "query types" queries

Where I am:

• I first tried querying with the listCaballos query adding the user input as a filter to the query, and using the nextToken recursively to go over the whole table (without using secondary indexes), but it took 6 minutes to go through the table and return the items.

• I know secondary indexes help to partition and then order the items through chosen keys (which makes it fast), buuuut I read that that forces the user to make an exact search (not a substring kind of search), and that's not what I need.

• I've heard Elastic Search might help.

Any suggestions?

Thanks!

Answer 1

This is not efficient in DynamoDB. Although you can create secondary indexes to search 'begins_with', substring ('contains') capability is there only for filters which are not efficient in a large data set (Since DynamoDB will use IOPS to query all and then apply the filter).

This kind of a requirement, it is efficient to index the database using another service like AWS ElasticSearch or CloudSearch so that you can apply the query on top of that service and configure continuous indexing.

Getting Started

• Searching DynamoDB Data with Amazon CloudSearch
• Combining DynamoDB and Amazon Elasticsearch with Lambda
• Indexing Amazon DynamoDB Content with Amazon Elasticsearch Service Using AWS Lambda