Building multiple executables with similar rules

Question

I am writing something like an interactive tutorial for C++. The tutorial will consist of two parts: one is compiled into a library (I'm using Scons to build that), and the other (the lessons) is shipped with the tutorial to be compiled by the end user. I'm currently looking for a good, easy way for people to build these lessons.

Basically, the second part is a directory with all the lessons in it, each in its own directory. Each lesson will have at least a lesson.cpp and a main.cpp file, there may be also other files, the existence of which I will not know until after it is shipped -- the end user will create these. It will look something like this:

all_lessons/
    helloworld/
        lesson.cpp
        main.cpp
    even_or_odd/
        lesson.cpp
        main.cpp
    calculator/
        lesson.cpp
        main.cpp
        user_created_add.cpp

Each of these will need to be compiled according to almost the same rules, and the command for compiling should be possible to run from one of the lesson directories (helloworld/, etc.).

Seeing as the rest of the project is built with Scons, it would make sense to use it for this part, too. However, Scons searches for the SConstruct file in the directory it is run from: would it be acceptable to put a SConstruct file in each lesson directory, plus a SConscript in the all_lessons/ directory that gives the general rules? This seems to go against the typical way Scons expects projects to be organised: what are the potential pitfalls of this approach? Could I put a SConstruct file instead of the SConscript one, and thereby make it possible to build from either directory (using exports to avoid endless recursion, I'm guessing)?

Also, I may at some point want to replace the lesson.cpp with a lesson.py that generates the necessary files; will Scons allow me to do this easily with builders, or is there a framework that would be more convenient?

In the end, I want to end up with the following (or equivalent with different build systems):

all_lessons/
    SConstruct
    helloworld/
        SConstruct
        lesson.cpp
        main.cpp
    even_or_odd/
        SConstruct
        lesson.py
        main.cpp
    calculator/
        SConstruct
        lesson.cpp
        main.cpp
        user_created_add.cpp

Running scons all in the all_lessons directory would need to:

• Run even_or_odd/lesson.py to generate even_or_odd/lesson.cpp.
• Realise that user_created_add.cpp also needs to be compiled.
• Produce an executable for each lesson.

Running scons in even_or_odd/, or scons even_or_odd in all_lessons/ should produce an executable identical to the one above (same compile flags).

Summary:

1. Is Scons suitable for/capable of this?
2. Does Scons work well when SConscript files are above SConstruct files?
3. Does Scons work well with multiple SConstrcut files for one project, SConscripting each other?
4. Is the Scons builder system suitable for using Python scripts to generate C++ files?
5. Is there any advantage of using a different build system/writing my own build framework that I'm missing?

Any further comments are, of course, welcome.

Thanks.

Answer 1

You can actually do this with a few lines of GNU Make.

Below are two makefiles that allow building and cleaning from all_lessons directory and individual project directories. It assumes that all C++ sources in that directory comprise an executable file which gets named after its directory. When building and cleaning from the top level source directory (all_lessons) it builds and cleans all the projects. When building and cleaning from a project's directory it only builds and cleans the project's binaries.

These makefiles also automatically generate dependencies and are fully parallelizable (make -j friendly).

For the following example I used the same source file structure as you have:

$ find all_lessons
all_lessons
all_lessons/even_or_odd
all_lessons/even_or_odd/main.cpp
all_lessons/Makefile
all_lessons/helloworld
all_lessons/helloworld/lesson.cpp
all_lessons/helloworld/main.cpp
all_lessons/project.mk
all_lessons/calculator
all_lessons/calculator/lesson.cpp
all_lessons/calculator/user_created_add.cpp
all_lessons/calculator/main.cpp

To be able to build from individial project directories project.mk must be symlinked as project/Makefile first

[all_lessons]$ cd all_lessons/calculator/
[calculator]$ ln -s ../project.mk Makefile
[helloworld]$ cd ../helloworld/
[helloworld]$ ln -s ../project.mk Makefile
[even_or_odd]$ cd ../even_or_odd/
[even_or_odd]$ ln -s ../project.mk Makefile

Let's build one project:

[even_or_odd]$ make
make -C .. project_dirs=even_or_odd all
make[1]: Entering directory `/home/max/src/all_lessons'
g++ -c -o even_or_odd/main.o -Wall -Wextra   -MD -MP -MF even_or_odd/main.d even_or_odd/main.cpp
g++ -o even_or_odd/even_or_odd even_or_odd/main.o  
make[1]: Leaving directory `/home/max/src/all_lessons'
[even_or_odd]$ ./even_or_odd
hello, even_or_odd

Now build all projects:

[even_or_odd]$ cd ..
[all_lessons]$ make
g++ -c -o calculator/lesson.o -Wall -Wextra   -MD -MP -MF calculator/lesson.d calculator/lesson.cpp
g++ -c -o calculator/user_created_add.o -Wall -Wextra   -MD -MP -MF calculator/user_created_add.d calculator/user_created_add.cpp
g++ -c -o calculator/main.o -Wall -Wextra   -MD -MP -MF calculator/main.d calculator/main.cpp
g++ -o calculator/calculator calculator/lesson.o calculator/user_created_add.o calculator/main.o  
g++ -c -o helloworld/lesson.o -Wall -Wextra   -MD -MP -MF helloworld/lesson.d helloworld/lesson.cpp
g++ -c -o helloworld/main.o -Wall -Wextra   -MD -MP -MF helloworld/main.d helloworld/main.cpp
g++ -o helloworld/helloworld helloworld/lesson.o helloworld/main.o  
[all_lessons]$ calculator/calculator 
hello, calculator
[all_lessons]$ helloworld/helloworld
hello, world

Clean one project:

[all_lessons]$ cd helloworld/
[helloworld]$ make clean
make -C .. project_dirs=helloworld clean
make[1]: Entering directory `/home/max/src/all_lessons'
rm -f helloworld/lesson.o helloworld/main.o helloworld/main.d helloworld/lesson.d helloworld/helloworld
make[1]: Leaving directory `/home/max/src/all_lessons'

Clean all projects:

[helloworld]$ cd ..
[all_lessons]$ make clean
rm -f calculator/lesson.o calculator/user_created_add.o calculator/main.o even_or_odd/main.o helloworld/lesson.o helloworld/main.o calculator/user_created_add.d calculator/main.d calculator/lesson.d even_or_odd/main.d  calculator/calculator even_or_odd/even_or_odd helloworld/helloworld

The makefiles:

[all_lessons]$ cat project.mk 
all :
% : forward_ # build any target by forwarding to the main makefile
    $(MAKE) -C .. project_dirs=$(notdir ${CURDIR}) $@
.PHONY : forward_

[all_lessons]$ cat Makefile 
# one directory per project, one executable per directory
project_dirs := $(shell find * -maxdepth 0 -type d )

# executables are named after its directory and go into the same directory
exes := $(foreach dir,${project_dirs},${dir}/${dir})

all : ${exes}

#  the rules

.SECONDEXPANSION:

objects = $(patsubst %.cpp,%.o,$(wildcard $(dir ${1})*.cpp))

# link
${exes} : % : $$(call objects,$$*) Makefile
    g++ -o $@ $(filter-out Makefile,$^) ${LDFLAGS} ${LDLIBS}

# compile .o and generate dependencies
%.o : %.cpp Makefile
    g++ -c -o $@ -Wall -Wextra ${CPPFLAGS} ${CXXFLAGS} -MD -MP -MF ${@:.o=.d} $<

.PHONY: clean

clean :
    rm -f $(foreach exe,${exes},$(call objects,${exe})) $(foreach dir,${project_dirs},$(wildcard ${dir}/*.d)) ${exes}

# include auto-generated dependency files
-include $(foreach dir,${project_dirs},$(wildcard ${dir}/*.d))