Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Where is the luigi config file?

Tags:

python

luigi

I have installed luigi by pip command and I would like to change the port for the web UI. I tried to find the config file but I couldn't. Do I need to create one?

like image 683
Iamasupernoob Avatar asked May 19 '17 08:05

Iamasupernoob


People also ask

Does Luigi have a scheduler?

By default, Luigi tasks run using the Luigi scheduler. To run one of your previous tasks using the Luigi scheduler omit the --local-scheduler argument from the command.

What is Luigi Python?

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

What are workers in Luigi?

worker module. The worker communicates with the scheduler and does two things: Sends all tasks that has to be run. Gets tasks from the scheduler that should be run.


2 Answers

You can start luigid with the --port option.

luigid --port 80

Configuration file locations are:

  • /etc/luigi/luigi.cfg
  • luigi.cfg (or its legacy name client.cfg) in your current working directory
  • LUIGI_CONFIG_PATH environment variable

in increasing order of preference. You do need to create one. e.g.,

[core]
default-scheduler-host=www.example.com
default-scheduler-port=8088
like image 116
MattMcKnight Avatar answered Oct 04 '22 07:10

MattMcKnight


In spite of the documentation saying otherwise, the port configuration from the config file is not used, at least in some versions or some circumstances

Until this is resolved, you should always use the --port option of luigid:

luigid --port 12345

Also see https://github.com/spotify/luigi/issues/2235

For other configuration options a config file should be used. See https://luigi.readthedocs.io/en/stable/configuration.html

For a configuration global to the host you can create a file:

/etc/luigi/luigi.cfg

Make sure it is readable by the user that runs luigid and luig.

Alternatively a local configuration file that will be recognized is

luigi.cfg

which you would have to create in the current working directory.

If you want a custom config file location you could set the environment variable LUIGI_CONFIG_PATH to the full path of your config file.

like image 26
mit Avatar answered Oct 04 '22 08:10

mit