Airflow 1.10 Config Core hostname_callable - How To Set?

Question

I'm regards to my previous Stackoverflow post here I've finally upgraded from Airflow version 1.9 to 1.10 since it's now released on PyPi. Using their release guide here I got Airflow 1.10 working. Now I inspected their udpates to 1.10 to see how they addressed the bug discovered in Airflow version 1.9 when run on an AWS EC2-Instance. And I found that they replaced all functions that got the servers IP address with a call to this new Airflow class's function get_hostname https://github.com/apache/incubator-airflow/blob/master/airflow/utils/net.py. Now, inside that small function you see the comment that says,

Fetch the hostname using the callable from the config or using socket.getfqdn as a fallback.

So then after that comment you see the code,

callable_path = conf.get('core', 'hostname_callable')

Which tells us in the airflow.cfg under the section [core] there is a new key value field called hostname_callable which now lets us set how we want to fetch the servers IP address. So their fix for the bug seen in Airflow version 1.9 is to just let us choose how to fetch the IP address if we need to change it. Their default value for this new configuration field is seen here https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg under the [core] section. You can see they have it set as,

[core]
# Hostname by providing a path to a callable, which will resolve the hostname
hostname_callable = socket:getfqdn

So they're using the function call socket:getfqdn that causes the bug to occur when run on an AWS EC2-Instance. I need it to use socket.gethostbyname(socket.gethostname()) (again this is mentioned in more detail on my previous post)

So my question is what's the syntax that I need to use in order to get socket.gethostbyname(socket.gethostname()) in the configuration style of using colons :. For example the function call socket.getfqdn() is written in the configuration file as socket:getfqdn. So I don't see how I would write that syntax with a nested call. Would I write something like socket:gethostbyname(socket.gethostname())?

Answer 1

If it were me, I would create an airflow_local_settings module with a hostname_callable function which returns the necessary value. From the code it looks like you could set the value to airflow_local_settings:hostname_callable.

Basically, in airflow_local_settings:

import socket

def hostname_callable():
  return socket.gethostbyname(socket.gethostname())

Then install your airflow_local_settings module to the computer as you would any other module.

Answer 2

This is not a direct answer but might help someone facing this or similar issues.

I ran into a similar issue when using Airflow locally on macOS Catalina. Sometimes (seemingly random) the local_task_job.py crashed saying

The recorded hostname mycustomhostname.local does not match this instance's hostname 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa

I could solve the issue by editing the airflow.cfg and replacing

hostname_callable = socket:getfqdn

with

hostname_callable = socket:gethostname