Calling a script after tasks queue is empty

Question

I am running a script which initiates several tasks:

for i in range (0,5):
  taskqueue.add(url='/example', 
                          params={'num': i})

As far as I understand, the tasks are running in parallel. Is there anyway I can tell AppEngine to run a specific task/python file once all the tasks I just inserted to the queue are ALL finished? I thought about sending a flag to the task that was called in the last loop iteration, but if tasks run in parallel it is not guarnteed that once it is finished, the other ones were finished too.

Thanks,

Joel

Answer 1

When you initiate the tasks, you know how many there will be. Insert an entity into the datastore with the 'expected' count, or list of tasks. Then use either counters or markers to indicate when a task has run. Roughly, the process might look something like:

The new kinds:

class TaskBatch(db.Model):
    expected_count = db.IntegerProperty()

class TaskMarker(db.Model):
    pass

Then, adjust your calling routine to do something like:

count = 5
taskbatch = TaskBatch(expected_count=count).put()
for i in range(5):
    taskqueue.add(url='/example',
                  params={'num': i, 'batch': str(taskbatch)})

And, at the end of your tasks:

def post(self):
    num = self.request.get('num')
    # do your stuff....

    batch = self.request.get('batch')
    TaskMarker(key_name=num, parent=db.Key(batch))
    taskqueue.add(url='/example/isdone',
                  params={'num': i, 'batch': str(taskbatch)})

And the isdone task could look something like:

def post(self):
    num = self.request.get('num')
    batch_key = db.Key(self.request.get('batch'))
    batch = TaskBatch.get(batch_key)
    marker_keys = [db.Key.from_path('TaskMarker', i, parent=batch)
                     for i in range(batch.expected_count)]
    markers = db.get(marker_keys)
    if all(markers):
       # do your done action.

The exact process will vary slightly depending on the specifics of your usecase, such as how many tasks your inserting.