appengine: cached reference property?

Question

How can I cache a Reference Property in Google App Engine?

For example, let's say I have the following models:

class Many(db.Model):
    few = db.ReferenceProperty(Few) 

class Few(db.Model):
    year = db.IntegerProperty()

Then I create many Many's that point to only one Few:

one_few = Few.get_or_insert(year=2009)
Many.get_or_insert(few=one_few)
Many.get_or_insert(few=one_few)
Many.get_or_insert(few=one_few)
Many.get_or_insert(few=one_few)
Many.get_or_insert(few=one_few)
Many.get_or_insert(few=one_few)

Now, if I want to iterate over all the Many's, reading their few value, I would do this:

for many in Many.all().fetch(1000):
  print "%s" % many.few.year

The question is:

• Will each access to many.few trigger a database lookup?
• If yes, is it possible to cache somewhere, as only one lookup should be enough to bring the same entity every time?

---

As noted in one comment: I know about memcache, but I'm not sure how I can "inject it" when I'm calling the other entity through a reference.

In any case memcache wouldn't be useful, as I need caching within an execution, not between them. Using memcache wouldn't help optimizing this call.

Answer 1

The first time you dereference any reference property, the entity is fetched - even if you'd previously fetched the same entity associated with a different reference property. This involves a datastore get operation, which isn't as expensive as a query, but is still worth avoiding if you can.

There's a good module that adds seamless caching of entities available here. It works at a lower level of the datastore, and will cache all datastore gets, not just dereferencing ReferenceProperties.

If you want to resolve a bunch of reference properties at once, there's another way: You can retrieve all the keys and fetch the entities in a single round trip, like so:

keys = [MyModel.ref.get_value_for_datastore(x) for x in referers]
referees = db.get(keys)

Finally, I've written a library that monkeypatches the db module to locally cache entities on a per-request basis (no memcache involved). It's available, here. One warning, though: It's got unit tests, but it's not widely used, so it could be broken.