I am trying to configure Luigi's retry mechanism so that failed tasks will be retried a few times. However, while the task is retried successfully, Luigi exits unsuccessfully:
===== Luigi Execution Summary =====
Scheduled 3 tasks of which:
* 2 ran successfully:
- 1 FailOnceThenSucceed(path=/tmp/job-id-18.subtask)
- 1 MasterTask(path=/tmp/job-id-18)
* 1 failed:
- 1 FailOnceThenSucceed(path=/tmp/job-id-18.subtask)
This progress looks :( because there were failed tasks
So the question is: how do I configure Luigi (I have installed version 2.3.3 with pip install) so that when a task fails once, but is then retried with success, then Luigi will exit successfully with This progress looks :)
instead of fail with This progress looks :(
?
Here is a minimal scheduler and worker config I've come up with, as well as tasks to demonstrate the behavior:
[scheduler]
retry_count = 3
retry-delay = 1
[worker]
keep_alive=true
mytasks.py:
import luigi
class FailOnceThenSucceed(luigi.Task):
path = luigi.Parameter()
def output(self):
return luigi.LocalTarget(self.path)
def run(self):
failmarker = luigi.LocalTarget(self.path + ".hasfailedonce")
if failmarker.exists():
with self.output().open('w') as target:
target.write('OK')
else:
with failmarker.open('w') as marker:
marker.write('Failed')
raise RuntimeError("Failed once")
class MasterTask(luigi.Task):
path = luigi.Parameter()
def requires(self):
return FailOnceThenSucceed(path=self.path + '.subtask')
def output(self):
return luigi.LocalTarget(self.path)
def run(self):
with self.output().open('w') as target:
target.write('OK')
Example execution:
PYTHONPATH=. luigi --module mytasks MasterTask --workers=2 --path='/tmp/job-id-18'
This is an old issue of Luigi - where successful retried tasks were not marked as such when failed and then succeeded on retry: https://github.com/spotify/luigi/issues/1932
It was fixed in version 2.7.2: https://github.com/spotify/luigi/releases/tag/2.7.2
I suggest you upgrade to the latest Luigi version, i.e. by running pip install -U luigi
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With