Azure Pipeline python script environment variables

Question

I have a python script which it has 2 line of code where I have my username and password and I don't want to push them to GitHub for obvious reasons, but they are important for my azure DevOps pipeline to run successfully. Following some documentation, I set in my python script the òs.environ.get` to be able to retrieve the value from a environment variable.

my code looks like this.

import os

usernameot = os.environ.get('USERNAMEOT')
passwordot = os.environ.get('PASSWORDOT')

wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class$='visible-lg'] input#signInFormUsername"))).send_keys(usernameot)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class$='visible-lg'] input#signInFormPassword"))).send_keys(passwordot)

This was the first step and after this problem started. I have a azure pipeline that looks like this.

trigger: none

pool:
  vmImage: 'ubuntu-latest'

stages:
  - stage:
    jobs:
      - job: Configuration
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '3.8'
              addToPath: true
          - script: |
              python -m pip install --upgrade pip
              pip install selenium
              printenv
          - task: Pythonscript@0
            inputs:
              scriptSource: 'filePath'
              scriptPath: './script1.py'
            env:
              USERNAMEOT: $(usernameot)
              PASSWORDOT: $(passwordot)

      - job: Mailinator
        dependsOn: Configuration
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '3.x'
              addToPath: true
          - script: |
              python -m pip install --upgrade pip
              pip install selenium
          - task: Pythonscript@0
            inputs:
              scriptSource: 'filePath'
              scriptPath: './script2.py'

I tried to pass the environment variable in all the ways that I know.

The pipeline above is the latest one, I tried to store the variables in azure DevOps pipeline but it fails as my python script doesn't find the username value in the environment variables. I tried to use GitHub secret and environment but it fails because it doesn't reconize the key secret.USERNAME

Can please any of you help me to understand how I can set a environment variable on my pipeline vm during the run time?

EDIT: I tried all the solutions advice by bazetto (thank you so much for your help) but still facing the same issue.

As you can see, the pipeline return the correct values for my variables, but those are not passed to the python script. Even if the error point to the css selector no found, I am pretty sure that the web driver is working because before to get to the username and password, there are different buttons to click etc. So what is my guess, is that when it comes to pass the username and password, as the variable does not get read from my configuration, the selenium script timeout.

Any advice about this please?

UPDATE: I did some update to the pipeline and my python script.

In python I am getting the username and configuration from a json file that get populated during the runtime, and I am feeding those variables values to my script.

following another advice, I set the variables on each step of my pipeline, but yet I am getting an error, as follow:

Traceback (most recent call last):
  File "D:\a\1\s\script1.py", line 49, in <module>
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class$='visible-lg'] input#signInFormUsername"))).send_keys(usernameot)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message


##[error]The process '/opt/hostedtoolcache/Python/3.8.11/x64/bin/python' failed with exit code 1

I really don't understand what I am doing wrong, I think I understand all the steps and what everyone is suggesting, but I might have miss something in my scripts and configuration for sure

Answer 1

When you define variables in Azure DevOps you have an option to mark them as secrets.

Above username is not a secrets, and password is. In this case not-secret variables are mapped to environment variables.

What you can check with this:

steps:
- script: env | sort
  displayName: 'Display env variables'

PATH=/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/vsts/.local/bin:/opt/pipx_bin:/usr/share/rust/.cargo/bin:/home/vsts/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/vsts/.dotnet/tools:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PIPELINE_WORKSPACE=/home/vsts/work/1
PIPX_BIN_DIR=/opt/pipx_bin
PIPX_HOME=/opt/pipx
.
.
.
TASK_DISPLAYNAME=Display env variables
TF_BUILD=True
USER=vsts
USERNAME=someusername
VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg

However, there is still a way how to map them to env variables. There is explicit mapping like below:

- bash: |
    echo "Using the mapped env var for this task works and is recommended: $MY_MAPPED_PASSWORD"
  env:
    MY_MAPPED_PASSWORD: $(password) 

But this is limited to task scope. So you can use this env variable only in this task. If you need this in more tasks, then you need to repeat that env mapping accross every task.

So if you have both username and password defined as secret you need to mapp both values:

          - task: Pythonscript@0
            inputs:
              scriptSource: 'filePath'
              scriptPath: './script1.py'
            env:
              PASSWORD: $(password) 
              USERNAME: $(username) 

And if you see *** in logs this is masking which Azure DevOps does to hide your secret if it appears anywhere in logs.

Answer 2

You might face issues related to environment variables due to admin privileges.

My suggestion is to replace it with a file, like

config.json

{
   "username": "",
   "password": ""
}

in your pipeline, add a step to populate it:

  - stage: temp

    jobs:
      - job: temp
        displayName: temp
        
        variables:
          - name: USERNAME
            value: "USER1"

          - name: PASSWORD
            value: "MYPWD1"
        
        steps:
          - checkout: self
        
          - task: PowerShell@2
            displayName: "Set configuration"
            inputs:
              targetType: "inline"
              script: |
                #read config
                $config = Get-Content (Get-Item .\config.json) -Raw -Encoding UTF8 | ConvertFrom-Json

                #set the credentials
                $config.username = "$(USERNAME)" #assuming that you already have it
                $config.password = "$(PASSWORD)" #assuming that you already have it


                #update the file
                $config | ConvertTo-Json | Set-Content .\config.json

          - task: PowerShell@2
            displayName: "show configuration"
            inputs:
              targetType: "inline"
              script: |
                #debug
                cat config.json

The configuration step must show something like:

in your python app something like:

import json

f = open('config.json',)
data = json.load(f)

username = data['username']
password = data['password']

#debug
print(data)