docker-compose Unknown MySQL server host 'mysql' (airflow)

Question

I try to create my Airflow service. When I run three dockers separately (without docker-compose) everything works but when I try to do the same using docker-compose I get the error. It seems that airflow doesn't see database.

docker-compose.yml

version: '2'

services:

  mysql:
    image: "someregisty/mysql"
    environment:
     - MYSQL_ROOT_PASSWORD=somepassword

  redis:
    image: "someregisty/redis"

  airflow:
    image: "someregisty/airflow"
    volumes:
     - ./dags:/home/airflow/dags
     - ./logs:/home/airflow/logs
     - ./config/airflow.cfg:/home/airflow/airflow.cfg 
     - ./config/resolv.conf:/etc/resolv.conf
     - ./config/somekeytab.keytab:/home/airflow/airflow.keytab 
    ports:
     - "8484:8080"
    environment:
     - AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
     - AIRFLOW__CELERY__CELERY_RESULT_BACKEND=db+mysql://root:somepassword@mysql/airflow
     - AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://root:somepassword@mysql/airflow
     - AIRFLOW__CORE__AIRFLOW_HOME=/home/airflow
     - AIRFLOW__CORE__DAGS_FOLDER=/home/airflow/dags/
     - AIRFLOW__CORE__PLUGINS_FOLDER=/home/airflow/plugins/
     - AIRFLOW_SCHEDULER=true
     - AIRFLOW_WEBSERVER=true
     - AIRFLOW_WORKER=true
     - AIRFLOW_KERBEROS=true
     - [email protected]
     - AIRFLOW__KERBEROS__CCACHE=/home/airflow/kerberos.keytab.cache
     - KRB5CCNAME=/home/airflow/kerberos.keytab.cache
    depends_on:
     - redis
     - mysql

I run

docker-compose up -d
docker-compose exec mysql mysql -u root --password=somepassword -e "create database airflow"

and that works ok. Then I run

docker-compose exec airflow airflow initdb

And I get

[2017-09-28 16:34:36,100] {__init__.py:57} INFO - Using executor CeleryExecutor
/usr/lib/python2.7/site-packages/airflow/utils/helpers.py:406: DeprecationWarning: Importing BaseSensorOperator directly from <module 'airflow.operators' from '/usr/lib/python2.7/site-packages/airflow/operators/__init__.pyc'> has been deprecated. Please import from '<module 'airflow.operators' from '/usr/lib/python2.7/site-packages/airflow/operators/__init__.pyc'>.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
  DeprecationWarning)
DB: mysql://root:***@mysql/airflow
[2017-09-28 16:34:36,666] {db.py:287} INFO - Creating tables
Traceback (most recent call last):
  File "/usr/bin/airflow", line 28, in <module>
    args.func(args)
  File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 951, in initdb
    db_utils.initdb()
  File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 106, in initdb
    upgradedb()
  File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 294, in upgradedb
    command.upgrade(config, 'heads')
  File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/airflow/migrations/env.py", line 86, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/site-packages/airflow/migrations/env.py", line 73, in run_migrations_online
    with connectable.connect() as connection:
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2091, in connect
    return self._connection_cls(self, **kwargs)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
    if connection is not None else engine.raw_connection()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection
    self.pool.unique_connection, _connection)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
    e, dialect, self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
    exc_info
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
    return fn()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 328, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
    rec = pool._do_get()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
    self._dec_overflow()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
    return self._create_connection()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2005, "Unknown MySQL server host 'mysql' (0)")

I don't know what is wrong. Maybe you could help me? :)

Versions of docker and docker-compose are

Docker version 1.12.6, build 96d83a5/1.12.6
docker-compose version 1.9.0, build 2585387

Answer 1

The problem is caused by binding resolv.conf file into airflow container. The reason you get unknown MySQL server host 'mysql' error from airflow container is that mysql hostname does not resolve to the ip of airflow container.

Solution 1

Remove ./config/resolv.conf:/etc/resolv.conf from Compose configuration. It will cause Docker to resolve hostname correctly. If you need to specify additional DNS servers use Compose --dns option documented here.

Solution 2 (not recommended)

Add nameserver 127.0.0.11 line to /etc/resolv.conf file. It enables usage of Docker embedded DNS server, which is a default setup. You can keep your custom entries in that file as well. Note that this solution is a hack because, as stated in documentation, Docker is responsible for managing /etc/resolv.conf file. Quote from documentation to back it up:

So you should not assume the way the files such as /etc/hosts, /etc/resolv.conf are managed inside the containers and leave the files alone and use the following Docker options instead

More

You can read more about Docker embedded DNS server in user-defined networks here.