how to dump gitlab ci environment variables to file

Question

the question

How to dump all Gitlab CI environment variables (with variables set in the project or group CI/CD settings) to a file, but only them, without environment variables of the host on which gitlab runner is executed?

Background

We are using gitlab CI/CD to deploy our projects to a docker server. Each project contains a docker-compose.yml file which uses various environment variables, eg db passwords. We are using .env file to store this variables, so one can start/restart the containers after deployment from command line, without accessing gitlab.

Our deployments script looks something like this:

deploy: 
  script:
    #...
    - cp docker-compose.development.yml {$DEPLOY_TO_PATH}/docker-compose.yml
    - env > variables.env
    - docker-compose up -d 
    #...

And the docker-compose.yml file looks like this:

version: "3"
services:
  project:
    image: some/image
    env_file:
      - variables.env
    ...

The problem is now the .env file contains both gitlab variables and hosts system environment variables and in the result the PATH variable is overwritten.

I have developed a workaround with grep:

env | grep -Pv "^PATH" > variables.env

It allowed us to keep this working for now, but I think that the problem might hit us again with another variables which would be set to different values inside a container and on the host system.

I know I can list all the variables in docker-compose and similar files, but we already have quite a few of them in a few projects so it is not a solution.

Answer 1

You need to add to script next command

script:
  ...
  # Read certificate stored in $KUBE_CA_PEM variable and save it in a new file
  - echo "$KUBE_CA_PEM" > variables.env
  ...

Answer 2

My reusable solution /tools/gitlab/script-gitlab-variables.yml:

variables:
  # Default values
  GITLAB_EXPORT_ENV_FILENAME: '.env.gitlab.cicd'

.script-gitlab-variables:
  debug:
    # section_start
    - echo -e "\e[0Ksection_start:`date +%s`:gitlab_variables_debug[collapsed=true]\r\e[0K[GITLAB VARIABLES DEBUG]"
    # command
    - env
    # section_end
    - echo -e "\e[0Ksection_end:`date +%s`:gitlab_variables_debug\r\e[0K"
  export-to-env:
    # section_start
    - echo -e "\e[0Ksection_start:`date +%s`:gitlab_variables_export_to_env[collapsed=true]\r\e[0K[GITLAB VARIABLES EXPORT]"
    # verify mandatory variables
    - test ! -z "$GITLAB_EXPORT_VARS" && echo "$GITLAB_EXPORT_VARS" || exit $?
    # display variables
    - echo "$GITLAB_EXPORT_ENV_FILENAME"
    # command
    - env | grep -E "^($GITLAB_EXPORT_VARS)=" > $GITLAB_EXPORT_ENV_FILENAME
    # section_end
    - echo -e "\e[0Ksection_end:`date +%s`:gitlab_variables_export_to_env\r\e[0K"
  cat-env:
    # section_start
    - echo -e "\e[0Ksection_start:`date +%s`:gitlab_variables_cat-env[collapsed=true]\r\e[0K[GITLAB VARIABLES CAT ENV]"
    # command
    - cat $GITLAB_EXPORT_ENV_FILENAME
    # section_end
    - echo -e "\e[0Ksection_end:`date +%s`:gitlab_variables_cat-env\r\e[0K"

How to use .gitlab-ci.yml:

include:
  - local: '/tools/gitlab/script-gitlab-variables.yml'

Your Job:
  variables:
    GITLAB_EXPORT_VARS: 'CI_BUILD_NAME|GITLAB_USER_NAME'
  script:
    - !reference [.script-gitlab-variables, debug]
    - !reference [.script-gitlab-variables, export-to-env]
    - !reference [.script-gitlab-variables, cat-env]

Result cat .env.gitlab.cicd:

CI_BUILD_NAME=Demo
GITLAB_USER_NAME=Benjamin

What you need dump all:

# /tools/gitlab/script-gitlab-variables.yml
  dump-all:
    - env > $GITLAB_EXPORT_ENV_FILENAME

# .gitlab-ci.yml
  script:
    - !reference [.script-gitlab-variables, dump-all]

I hope I could help