Google App Engine Datastore index cap

Question

Can somebody explain the 5000 index cap in the Datastore in plain English.
Does that mean that an indexed list property of a stored object cannot have more the 5000 elements?

Answer 1

The Datastore limits the number of index entries that a single entity can have, this limit is set to 5000 elements per entity.

You can test this limit easily using the Interactive shell with the following snippet:

class Model(db.Model):
   x = db.ListProperty(int)

entity = Model(x = range(5001))
entity.put()

'Too many indexed properties for entity %r.' % self.key())
BadRequestError: Too many indexed properties for entity  
datastore_types.Key.from_path(u'Model', 0, _app=u'shell')

Answer 2

Short answer, Yes if you indexed the property.

App Engine limits the number of property values a single entity can have on an index (number of rows * the number of columns) since it needs to create an index for each permutation. In the case of a single index property you have 5000rows * 1column = 5000.

To show why App Engine does this let's take the example from their documentation.

Model:

class MyModel(db.Model):
   x = db.StringListProperty()
   y = db.StringListProperty()

Index.yaml

indexes:
- kind: MyModel
  properties:
  - name: x
  - name: y

Execution

e2 = MyModel()
e2.x = ['red', 'blue']
e2.y = [1, 2]
e2.put()

In this case, App Engine will have to create 12 indexes for this datastore entry alone, since effectively you can search for any combination of values:

x1
x2
y1
y2
x1 y1
x1 y2
x2 y1
x2 y2
y1 x1
y1 x2
y2 x1
y2 x2

Now if you had 100 values in each property, you can imagine that the list would sky rocket to an obscene amount of queries.

The equation is something like this:

len(x) + len(y) + (len(x)-1 * len(y) * (len(x) + len(y))) = number of indexed

**2 values per property**
2 + 2 + (1 * 2 * (2 + 2)) = 12

**100 values per property**
100 + 100 + (99 * 100 * (100 + 100)) = 1,980,200