Invalid Fernet key when an existing yet exited airflow container is again started

Question

I Have dockerized airflow service with a FERNET_KEY as an environment variable exported using a .env file(which also includes other environment variables apart from FERNET_KEY).

When I build and run the docker container, the airflow service runs as expected and then I go to connections and set up my AWS connections in the airflow UI as mentioned here https://airflow.apache.org/howto/connection/aws.html . So far so good. Now, I deliberately stop the docker container and then start the container again and go to airflow UI--> Connections to see if my AWS connection settings are still there(which I configured before stopping the container).

But, here is what happens. when I go to see aws_default settings in airflow UI, I see an error as below

Ooops.

                      ____/ (  (    )   )  \___
                     /( (  (  )   _    ))  )   )\
                   ((     (   )(    )  )   (   )  )
                 ((/  ( _(   )   (   _) ) (  () )  )
                ( (  ( (_)   ((    (   )  .((_ ) .  )_
               ( (  )    (      (  )    )   ) . ) (   )
              (  (   (  (   ) (  _  ( _) ).  ) . ) ) ( )
              ( (  (   ) (  )   (  ))     ) _)(   )  )  )
             ( (  ( \ ) (    (_  ( ) ( )  )   ) )  )) ( )
              (  (   (  (   (_ ( ) ( _    )  ) (  )  )   )
             ( (  ( (  (  )     (_  )  ) )  _)   ) _( ( )
              ((  (   )(    (     _    )   _) _(_ (  (_ )
               (_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
               ((__)        \\||lll|l||///          \_))
                        (   /(/ (  )  ) )\   )
                      (    ( ( ( | | ) ) )\   )
                       (   /(| / ( )) ) ) )) )
                     (     ( ((((_(|)_)))))     )
                      (      ||\(|(|)|/||     )
                    (        |(||(||)||||        )
                      (     //|/l|||)|\\ \     )
                    (/ / //  /|//||||\\  \ \  \ _)
-------------------------------------------------------------------------------
Node: 64e7a509837f
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 2125, in edit_view
    form = self.edit_form(obj=model)
  File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 1340, in edit_form
    return self._edit_form_class(get_form_data(), obj=obj)
  File "/usr/local/lib/python3.7/site-packages/wtforms/form.py", line 212, in __call__
    return type.__call__(cls, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask_admin/form/__init__.py", line 16, in __init__
    super(BaseForm, self).__init__(formdata=formdata, obj=obj, prefix=prefix, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/wtforms/form.py", line 278, in __init__
    self.process(formdata, obj, data=data, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/wtforms/form.py", line 127, in process
    if obj is not None and hasattr(obj, name):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/attributes.py", line 353, in __get__
    retval = self.descriptor.__get__(instance, owner)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 155, in get_password
    return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
  File "/usr/local/lib/python3.7/site-packages/cryptography/fernet.py", line 171, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken

I may be missing some concept about Fernet keys. Do I have to generate new fernet key every single time my docker container starts? If so, how should I generate on the fly when docker is building? The FERNET_KEY which I injected in to .env file is actually generated in my local computer and copied over to .env file(FERNET_KEY=4EPOSLXXXXXXXXXXXIERu=) which docker is using

Answer 1

You shouldn't need to generate a new key on the fly. You maybe missing something, have you tried to connect to your docker machine and print the FERNET_KEY from the terminal to check if it is not really been loaded?

$ docker exec -it <CONTAINER ID> bash
(now, inside the container) # echo ${FERNET_KEY}

If it is not the same as your env file there is something wrong. Are you using a docker-compose environment? If so, you also need to the following to docker-compose.yml:

environment:
    - FERNET_KEY=${FERNET_KEY}

Answer 2

I had the same issue. I'm using a mac and running the containers with docker-compose.

I solved it by creating a new FERNET_KEY

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

and adding the line with the new generated key (including the final "=" to) to my ~/.bash_profile file in this way: export FERNET_KEY="new_generated_key".

After that, I added

environment:
    - FERNET_KEY=${FERNET_KEY}

to my docker-compose (under the service of the webserver of course), like Renato was saying above.