Is it possible to "pass-through" GNU make jobserver environment to a submake served via a 3rd-party (non-make)

Question

When running gnu-make rules with -jN make creates a jobserver for managing job-count across submakes. Additionally you can "pass the jobserver environment" to a make recipe by prefixing it with + - eg:

target :
        +./some/complex/call/to/another/make target

Now I instead of a sub-make I have a (python) script which runs some complex packaging actions (too complex for make). One of the actions that it can run into can actually spawn off a make command.

package.stamp : $(DEPS)
         +./packaging.py $(ARGS)
         touch $@

Now when that make command is invoked inside packaging.py

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule.

This makes some sense because whatever environment is setup by make, may not be being honoured or passed through by python.

Is it possible to pass through the jobserver references through the python program to the sub-make - if so, how?

Answer 1

There are two aspects to the jobserver that must be preserved: the first is an actual environment variable, which make uses to send options to sub-makes. That value is being preserved properly, or else make would not know that it should even look for the jobserver and you would not see that warning message.

The second aspect are two open file descriptors which are passed to the children of make. Your script MUST preserve these two descriptors and leave them open when it invokes the sub-make.

You don't show us what Python code is being used to invoke the sub-make. By default, the subprocess module will not close file descriptors, but you can provide the close_fds=True option to have it do so... you should not use this option if you want parallel make invocations to work properly with the jobserver.

If you're not using subprocess, then you'll have to show us what you are doing.

You should probably mark this with a python tag as it's mainly a Python question.