How to retry celery task on hard timeout?

Question

I have a celery task as follows:

@celery.task(name='tasks.ht_run', default_retry_delay=30, max_retries=15, time_limit=1)  
def ht_run(str_command):
    try:
        f = os.popen(str_command)
        output = f.read()
        if output == '':
            raise Exception
    except Exception as exc:
        raise ht_run.retry(exc=exc)

    return output.split('\n')

And it is called like so: appserver.ht_run.delay(string)

While I expect it to retry when it timeouts, instead if just fails. In the Celery window I get the following error:

[2014-12-10 11:50:22,128: ERROR/MainProcess] Task tasks.ht_run[6a83793a-8bd6-47fc-bf74-0b673bf961f2] raised unexpected: TimeLimitExceeded(1,)
Traceback (most recent call last):
  File "/Users/andy.terhune/cgenv/lib/python2.6/site-packages/billiard/pool.py", line 639, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
TimeLimitExceeded: TimeLimitExceeded(1,)
[2014-12-10 11:50:22,128: ERROR/MainProcess] Hard time limit (1s) exceeded for tasks.ht_run[6a83793a-8bd6-47fc-bf74-0b673bf961f2]
[2014-12-10 11:50:23,644: ERROR/MainProcess] Process 'Worker-28' pid:2081 exited with 'signal 9 (SIGKILL)'

How can I get this to time out and retry when it does?

Answer 1

Thanks user2097159,

tried a soft timeout and worked like a charm,

@celery.task(
    name='tasks.ht_run', 
    default_retry_delay=30,
    max_retries=15,
    soft_time_limit=1)
def ht_run(str_command):
    try:
        f = os.popen(str_command)
        output = f.read()
        if output == '':
            raise Exception
    except Exception as exc:
        raise ht_run.retry(exc=exc)

    return output.split('\n')

...

[2014-12-10 12:16:55,580: WARNING/MainProcess] Soft time limit (1s) exceeded for tasks.ht_run[f31bbd15-e755-440c-9261-cd0864ceb3a9]
[2014-12-10 12:16:55,603: INFO/MainProcess] Received task: tasks.ht_run[f31bbd15-e755-440c-9261-cd0864ceb3a9] eta:[2014-12-10 18:17:25.593613+00:00]
[2014-12-10 12:16:55,603: DEBUG/MainProcess] basic.qos: prefetch_count->121
[2014-12-10 12:16:56,897: INFO/MainProcess] Task tasks.ht_run[f31bbd15-e755-440c-9261-cd0864ceb3a9] retry: Retry in 30s: SoftTimeLimitExceeded()