Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Celery chain breaks if one of the tasks fail

What is the point of Celery chain if the whole chain breaks if one of the tasks fail?!!

I have this Celery chain:

res = chain(workme.s ( y=1111 ), workme2.s( 2222 ), workme3.s( 3333 ),)() 

And I made workme2 fails with retries like this:

@celery.task(default_retry_delay=5, max_retries = 10, queue="sure") 
def workme2(x,y):
    # try:      
    try:
        print str(y)
        sleep(2)
        print str(x)
        ## adding any condition that makes the task fail
        if x!=None:
            raise Exception('Aproblem from your workme task')
        print 'This is my username: ' + str(x['user']) + \
               ' And Password: ' + str(x['pas'])        
        return "22xx"
    except Exception, exc:
        workme2.retry(args=[x,y], exc=exc,)
like image 389
securecurve Avatar asked Jun 25 '13 07:06

securecurve


1 Answers

That is the point.

Forming a chain means your subtasks have some kind of serial dependency: Each one only makes sense if the previous one has been performed. Without this, you would simply use queueing or use a group rather than a chain.

So if one subtask fails (and still fails after attempting all its retries), the chain fails.

I readily admit that the documentation (as of Celery 3.1.18) is far from explicit in this respect, but the name suggests this semantics: "A chain is only as strong as its weakest link."

like image 72
Lutz Prechelt Avatar answered Oct 01 '22 20:10

Lutz Prechelt