What to do when I don't want Luigi to output a file but show the task as complete?

Question

When looping over files with Luigi I do not what to be forced to save empty files just to show that the task was complete, and let the next task check if there are any rows in the txt, etc.

How can I have a task showing it succeeded (i.e. the run method worked as expected) without outputting a file? Am I missing something here?

Answer 1

You can override the complete function.

class LuigiTaskB(luigi.Task):
    def run(self):
        print "running task b"
        with self.output().open('w') as out_file:
            print >> out_file, "some text"

    def output(self):
        return luigi.LocalTarget("somefile")


class LuigiTaskA(luigi.Task):
    task_complete = False
    def requires(self):
        return LuigiTaskB()

    def run(self):
        print "running task a"
        self.task_complete = true

    def complete(self):
        # Make sure you return false when you want the task to run.
        # And true when complete

        return  self.task_complete
        # This will out put :
        #   running task b
        #   running task a
        # And this on the second time you'll run:
        #   running task a

The complete() function is looking at the output() function, by overriding complete() you can pass on any output and write your on complete condition.

Notice that if your complete function depends on the run function it may not be skipped..