Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

In the python version of Google App Engine mapreduce, how do you access counters from the done_callback?

I am using Google App Engine mapreduce to analyze some data. I am generating a few counters that I would like to create a simple Google chart from in my done_callback. How do I access the resulting counters from the callback?

#The map method
def count_created_since(entity):
  now = datetime.datetime.now()
  delta = now-entity.created

  #analyze last 12 weeks
  for x in range(12):
    start = 7*x
    stop = 7*(x+1)

    if delta.days >= start and delta.days < stop:
      #The counters
      yield op.counters.Increment(str(x)+" weeks ago")


def my_callback(request):
  # fetch counter results to create a simple Google chart url
like image 577
Chris Avatar asked Oct 25 '10 02:10

Chris


People also ask

What is MapReduce in Google App Engine?

MapReduce is a programming model for processing large amounts of data in a parallel and distributed fashion. It is useful for large, long-running jobs that cannot be handled within the scope of a single request, tasks like: Analyzing application logs. Aggregating related data from external sources.

How does Google App Engine Works?

App Engine is a fully managed, serverless platform for developing and hosting web applications at scale. You can choose from several popular languages, libraries, and frameworks to develop your apps, and then let App Engine take care of provisioning servers and scaling your app instances based on demand.

What are the services of Google App Engine?

From App Engine, you can easily access other Google Cloud services such as Datastore, Cloud SQL, and Cloud Storage. You also have the option to use an external or third-party database if that database is supported by your language and accessible from your App Engine instance.


1 Answers

You can access the counter's through a MapreduceState's counter_map attribute.

from mapreduce import model
state = model.MapreduceState.get_by_job_id(your_job_id)
# counters live in state.counters_map

There was a discussion on the mailing list a month or so ago about accessing counters.

like image 159
Robert Kluin Avatar answered Sep 24 '22 20:09

Robert Kluin