How to log from Python Luigi

Question

I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.

Answer 1

Inside any of the Task class methods, you can do:

class Agg(luigi.Task):
  _date = luigi.DateParameter()

  def output(self):
    return luigi.LocalTarget("file_%.txt" % self._date)

  def run(self):
    # Use the luigi-interface to log to console
    logger = logging.getLogger('luigi-interface')
    logger.info("Running --> Agg.Task")