SCons dependency on files generated at compile-time

Question

I'm trying to set up SCons to follow dependencies on files that are generated automatically during the build, and work correctly with a multi-threaded build.

The project I'm building is a CIM provider, consisting of MOF files defining the data structures, autogenerated source and header files that come from the MOF files, and handwritten source and header files that reference the autogenerated files. In order for the build to succeed, the autogeneration step must run to completion before any of the handwritten files are compiled, otherwise the headers the handwritten files rely on will not exist yet and it will fail. The .cpp files that are created by the autogeneration step must also be added to the source list and compiled in the final build.

When running a single threaded build, everything works fine, because the autogeneration step is always complete before the compilation step, so the generated headers are in place. However, when running the build multi-threaded, it tries to compile the handwritten files before the autogeneration step has finished, and the build fails. I have specified an explicit dependency but SCons is not following it.

Here is the relevant portion of my SConscript file, I removed individual filenames from cim_targets[] because the list is VERY long, but to summarize, cim_targets[] is the list of target output files for the autogeneration step, provider_sources[] is the returned list of sources after the autogen step completes, sources[] is the list of handwritten source files, GenProvider() is a Command builder defined externally that executes the autogeneration step, and SharedLibrary() is an externally defined builder that does what it sounds like, uses the SCons library builder with a few extensions

# Define directory paths for the CIM schema
cim_dir = 'cim-schema-2.26.0'

var_smis_dir   = Dir('.').abspath # src/lib/XXX in variant

cim_sources = [
    Glob(os.path.join(cim_dir, '*qualifiers*.mof')),
    Glob(os.path.join(cim_dir, 'Core')     + '/CIM_*.mof'),
    Glob(os.path.join(cim_dir, 'Device')   + '/CIM_*.mof'),
    Glob(os.path.join(cim_dir, 'Event')    + '/CIM_*.mof'),
    Glob(os.path.join(cim_dir, 'XXXXXX')   + '/XXX_*.mof'),
    Glob(os.path.join(cim_dir, 'Interop')  + '/CIM_*.mof'),
    Glob(os.path.join(cim_dir, 'Physical') + '/CIM_*.mof'),
    Glob(os.path.join(cim_dir, 'System')   + '/CIM_*.mof'),
]

cim_sources_flat = []
for cim in cim_sources:
    for src in cim:
        cim_sources_flat.append(src)

cim_targets = [
    ......
]

sources = [
    'driver.cpp',
    'device.cpp',
    'cim_static_data.cpp',
    'module.cpp',
    'diag_log.cpp',
    'profile_element.cpp',
]

staticlibs = [
    ......
    ]


dynamiclibs = [
    .....
    ]

var_cim_sources = this_env.Install(var_smis_dir, cim_sources_flat)

cim_mof = 'cimv226.mof'

cim_linux_mof = os.path.join(cim_dir, 'cimv226-gen-flat.mof')

var_cim_sources.extend(this_env.Command(cim_mof, cim_linux_mof, Copy('$TARGET', '$SOURCE')))

# first generate the provider infrastructure using cimple
provider_sources = this_env.GenProvider(cim_targets, var_cim_sources, name, var_smis_dir)

# make sure these files don't build until AFTER the provider files have been created
this_env.Depends(sources, provider_sources)

sources_full = provider_sources + sources

# now we can compile the provider
this_env.SharedLibrary(libname, source=sources_full, staticlibs=staticlibs, dynamiclibs=dynamiclibs, installpath=install_dir)

I tried setting an explicit dependency so that the handwritten sources would not compile until all generated sources were created (this_env.Depends(sources, provider_sources)) but when running multi-threaded, SCons ignores this dependency and tries to compile the handwritten files before the autogeneration step has completed.

Answer 1

Have you tried using the SideEffect() function as defined here:

SCons Wiki: SideEffect

Im not sure if it was created exactly for your need, but may help.