Update entity from a sharded counter - Python

Question

In my appengine python app, I have sharded counters that count the number of favs for a photo, and number of views.

I have Photo and Counter models. To order photos by popularity (# of favs), I should have the # of favs stored in my Photo entities (Photo.all().order('num_favs')). Since I'm keeping track of the num of favs in a sharded counter, I'm wondering what's the most efficient way to update my Photo entities with the latest # of favs from the Counter model?

Here's an abstract of how my models look like:

class Photo(db.Model):
  photo = db.BlobProperty() 
  favs = db.IntegerProperty() #num of favs

class Counter(db.Model):
  #key_name = key to a model that's to be counted - Photo in this case
  name =  db.StringProperty() #counted entity key
  count = db.IntegerProperty() #count
  type = db.StringProperty() #type of model to count, "Photo" in this example

#Very unefficient way of updating my Photo entities with the latest count from Counter:

fav_counters = Counter.all().filter('type =', 'Photo')

for fav in fav_counters:
  photo_fav = db.get(fav.name) #get the photo for which the fav count is for
  photo_fav.favs = fav.count
  photo_fav.put()

I would appreciate it if I could answers on: 1) The most efficient way to order entities based on their counters 2) If the logic I'm following to order entities based on their counters is correct - what's the most efficient way to update the Photo entities with their counters?

Thank you!

Answer 1

If you need to sort or filter based on a value, you probably shouldn't use a sharded counter, but rather use one of the alternatives. In short, they are:

1. Simply use a regular counter. If you don't expect the rate of updates to exceed 1-5QPS for extended periods (brief spikes are okay), then this should work fine.
2. Put a task in the task queue when a user favorites or un-favorites a photo. Ensure the task queue has a limited execution rate to limit contention.
3. Use one of the lossy but contention-free counter patterns, such as this one.