What is the best way to do AppEngine Model Memcaching?

Question

Currently my application caches models in memcache like this:

memcache.set("somekey", aModel)

But Nicks' post at http://blog.notdot.net/2009/9/Efficient-model-memcaching suggests that first converting it to protobuffers is a lot more efficient. But after running some tests I found out it's indeed smaller in size, but actually slower (~10%).

Do others have the same experience or am I doing something wrong?

Test results: http://1.latest.sofatest.appspot.com/?times=1000

import pickle
import time
import uuid

from google.appengine.ext import webapp
from google.appengine.ext import db
from google.appengine.ext.webapp import util
from google.appengine.datastore import entity_pb
from google.appengine.api import memcache

class Person(db.Model):
 name = db.StringProperty()

times = 10000

class MainHandler(webapp.RequestHandler):

 def get(self):

  self.response.headers['Content-Type'] = 'text/plain'

  m = Person(name='Koen Bok')

  t1 = time.time()

  for i in xrange(int(self.request.get('times', 1))):
   key = uuid.uuid4().hex
   memcache.set(key, m)
   r = memcache.get(key)

  self.response.out.write('Pickle took: %.2f' % (time.time() - t1))


  t1 = time.time()

  for i in xrange(int(self.request.get('times', 1))):
   key = uuid.uuid4().hex
   memcache.set(key, db.model_to_protobuf(m).Encode())
   r = db.model_from_protobuf(entity_pb.EntityProto(memcache.get(key)))


  self.response.out.write('Proto took: %.2f' % (time.time() - t1))


def main():
 application = webapp.WSGIApplication([('/', MainHandler)], debug=True)
 util.run_wsgi_app(application)


if __name__ == '__main__':
 main()

Answer 1

The Memcache call still pickles the object with or without using protobuf. Pickle is faster with a protobuf object since it has a very simple model

Plain pickle objects are larger than protobuf+pickle objects, hence they save time on Memcache, but there is more processor time in doing the protobuf conversion

Therefore in general either method works out about the same...but

The reason you should use protobuf is it can handle changes between versions of the models, whereas Pickle will error. This problem will bite you one day, so best to handle it sooner

Answer 2

Both pickle and protobufs are slow in App Engine since they're implemented in pure Python. I've found that writing my own, simple serialization code using methods like str.join tends to be faster since most of the work is done in C. But that only works for simple datatypes.