Use the Datastore (NDB), the Search API or both for views on data?

Question

In a CMS, a list of customers is retrieved using a regular NDB query with ordering. To allow filtering on name, company name and email, I create several (sometimes many) indices. The situation was not ideal, but workable.

Now there's the (experimental) Search API. It seems to have no relation to the datastore (or NDB), but my data is already there.

I'd like to use Full Text Search and put filters on multiple fields simultaniously, so should I keep my data in the Datastore and duplicate parts of the data in Documents for the Search API? Or, as the search example suggests, skip the Datastore entirely.

Answer 1

I'm not quite sure what the recommended method is for its implementation, but the Search API seems designed to be utilized primarily as an additional, manually managed index. Its not ideal in most situations to store all of your data in the Search API, as you can easily blow out the size of the search API indexes with fields which you never need to filter or search, nor is it well designed to be utilized in situation where regular writes are necessary.

My personal recommendation is to leave all of your data in NDB, and design classes to create documents containing the relevant searchable data using the Search API, maintaining consistency between the two medium's by updating the Search API version each time a write is made to the datastore version (or using tasks/crons or a similar system if you're writing data a lot). You should store any data you present in the UI when filtering in the Search API documents for that related data, as manually joining the search API results and the datastore data is unnecessarily intensive, and will severely slow down the whole process.