How can you "source" an environment inside a Buildbot step?

Question

Within Buildbot I need to be able to "source" an environment before doing a compilation step.

If I was building the application from command line using bash I would have to do:

. envrionment-set-up-script
build_command

Within the build bot master.cfg file I have tried the following:

factory.addStep(ShellCommand(command=["source","environment-set-up-script"])
factory.addStep(ShellCommand(command=[".","environment-set-up-script"]))
factory.addStep(Configure(command=["source","environment-set-up-script"]))
factory.addStep(Configure(command=[".","environment-set-up-script"]))

All of which fail, this is because the command cannot be found, which makes sense as it is a bash builtin.

Also I do not think that this is the correct approach as the environment would not necessarily be used when the next step of the factory is called.

Answer 1

After some experimenting I have found a way in which to achieve this. You need to:

• run a bash sub-shell setting the environment that should be used for that shell, i.e. call bash with the environment variable BASH_ENV set to the file that should be sourced into the environment.
• run the env command in bash to capture the environment
• parse the result of the env command into a property (using a SetProperty step)
• use the property within further steps as the env parameter

Note: that the environment should be parsed as a dictionary that can be used as an env parameter

    from buildbot.process.factory import BuildFactory
    from buildbot.steps.shell import ShellCommand, SetProperty
    from buildbot.process.properties import Property  

    def glob2list(rc, stdout, stderr):
        ''' Function used as the extrat_fn function for SetProperty class
            This takes the output from env command and creates a dictionary of 
            the environment, the result of which is stored in a property names
            env'''
        if not rc:
            env_list = [ l.strip() for l in stdout.split('\n') ]
            env_dict={ l.split('=',1)[0]:l.split('=',1)[1] for l in 
                          env_list if len(l.split('=',1))==2}
            return {'env':env_dict}

    #This is the equivalent of running source MyWorkdir/my-shell-script then  
    #capturing the environment afterwords.
    factory.addStep(SetProperty(command="bash -c env",
                extract_fn=glob2list,       
                workdir='MyWorkdir',
                env={BASH_ENV':'my-shell-script' }))

    #Do another step with the environment that was sourced from 
    #MyWorkdir/my-shell-script
    factory.addStep(ShellCommand(command=["env"],
                workdir="MyWorkdir",
                env=Property('env')))

Answer 2

An example for Visual Studio development is also useful.

By using common tools scripts, the right env is set for each machine, multiple versions of VS can be used by the same builders, and multiple architectures native and cross are supported.

# A raw text string to get around windows quoting problems in buildbot.
vscomntools=r""""%VS120COMNTOOLS%\VsDevCmd.bat" %PROCESSOR_ARCHITECTURE% & set"""

# Determine MSVC Environment using VS Common Tools for build steps.
f.addStep(SetProperty(
    command=vscomntools,
    extract_fn=glob2list))

The alternative is to start each command by trying to quote the batch file, &, and command.

Answer 3

While working with OpenEmbedded/Yocto, we solved the problem in a way similar to this:

class EnvStep(ShellCommand):
    def __init__(self, command='', **kwargs):
        kwargs['command'] = [
            'bash', '-c', 'source environment-set-up-script; %s' % command
        ]
        ShellCommand.__init__(self, **kwargs)

Then, adding an EnvStep that sets its command parameter to foo lets us run foo within the environment sourced with environment-set-up-script. In other words, you would use the step by calling

factory.addStep(EnvStep(command='foo'))

and the sourcing would happen automagically.

We also have a slew of other custom buildsteps that require the build-env to be sourced, so we just let them subclass EnvStep instead of ShellCommand for the environment to be dealt with automatically.