Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to make a Parameter available to all Luigi Tasks?

Tags:

python

luigi

In the Luigi docs, the use of a luigi.Config class is recommended for global configuration.

However, I am running into issues when using such a config class in order to pass a commandline argument to various Tasks in the pipeline.
Here's a lightweight example:

import datetime
import luigi


class HelloWorldTask(luigi.Task):

    def run(self):
        print("{task} says: Hello world on {date}!".format(task=self.__class__.__name__,
                                                           date=GlobalParams.date.strftime('%d-%b-%Y')))


class GlobalParams(luigi.Config):
    date = luigi.DateParameter(default=datetime.date.today())


if __name__ == '__main__':
    luigi.run(['HelloWorldTask', '--workers', '1', '--local-scheduler',
               '--GlobalParams-date', '2018-01-01'])

The class GlobalParams defines a DateParameter which I would like to later reference in the run() blocks of pipeline Tasks. However, this fails with the error, AttributeError: 'DateParameter' object has no attribute 'strftime'.

In the debugger, I can see that a DateParameter object is passed to the HelloWorldTask Task, but any attempts to extract the expected '2018-01-01' value passed at runtime fails.

Am I misunderstanding how to use these constructs? How should I be passing a single parameter to (possibly many) Tasks?

like image 769
Librarian Avatar asked Jan 29 '18 20:01

Librarian


1 Answers

The issue in the example code is that the GlobalParams class is not being instantiated before its parameter is accessed.
GlobalParams.date.strftime('%d-%b-%Y') should read GlobalParams().date.strftime('%d-%b-%Y').

This is included, but easy to overlook, in the configuration docs linked in the question.

like image 132
Librarian Avatar answered Oct 25 '22 02:10

Librarian