How to get QMake to copy large data files only if they are updated

Question

I have some large data files that need to be copied from source folders to build folders during our Qmake/QtCreator build. Since they are large, I only want the copy to happen for new/changed files. And I'd really like to avoid listing them all specifically in the project file. Here's what I've tried:

This attempt at copying data files fails because the DemoData folder is the target. Therefore the copy is not performed if files within the folder are added or changed. Only if the folder does not exist.

DemoData.commands = $$COPY_CMD $${SRC_DATA_DIR}DemoData $${BLD_DATA_DIR}DemoData
DemoData.target += $${BLD_DATA_DIR}DemoData
PRE_TARGETDEPS += $${BLD_DATA_DIR}DemoData
QMAKE_EXTRA_TARGETS += DemoData

This approach fails because the DemoData.target item is not expected to have a list of multiple items. QMake puts the list in quotes in the generated makefile so it becomes one target.

DemoData.commands = $$COPY_CMD $${SRC_DATA_DIR}DemoData $${BLD_DATA_DIR}DemoData
DEMO_DATA_FILES = $$files($${SRC_DATA_DIR}DemoData/*)
for(FILE, DEMO_DATA_FILES){
    DemoData.target += $${BLD_DATA_DIR}DemoData\\$$basename(FILE)
    PRE_TARGETDEPS += $${BLD_DATA_DIR}DemoData\\$$basename(FILE)
}
QMAKE_EXTRA_TARGETS += DemoData

This attempt fails because (AFAICT) QMake does not support variable names contained in other variables. It seems to be more of a one level substitution. A makefile is generated, but the DemoDataX targets all have no command lines. All attempts to display the contents of the 'commands' field generate syntax errors.

DEMO_DATA_FILES = $$files($${SRC_DATA_DIR}DemoData/*)
DEMO_DATA_NAME = DemoData
for(FILE, DEMO_DATA_FILES){
    $${DEMO_DATA_NAME}.target = $${FILE}
    $${DEMO_DATA_NAME}.commands = $$COPY_CMD $${FILE} $${BLD_DATA_DIR}DemoData
    PRE_TARGETDEPS += $${FILE}
    QMAKE_EXTRA_TARGETS += $${DEMO_DATA_NAME}
    DEMO_DATA_NAME = $${DEMO_DATA_NAME}X
}

This approach works, but with two shortcomings. The minor one is that a separate 'make install' step must be performed. The major one is that the files are always copied unconditionally. Since our data files are large, this is unacceptable timewise.

DemoData.path = $${BLD_DATA_DIR}DemoData
DemoData.files = $${SRC_DATA_DIR}DemoData/*
INSTALLS += DemoData

Is there a way to do this, or am I left with some sort of external script or manually generated/maintained makefile?

Answer 1

Use QMAKE_EXTRA_COMPILES feature.

# list your files in this variable.
# Masks are available with $$files functions but
# if your set of files changes (files added or removed)
# your have to re-run qmake after that explicitly, not just make
MYFILES = $$files($${PWD}/files/*.*)

copy_files.name = copy large files
copy_files.input = MYFILES
# change datafiles to a directory you want to put the files to
copy_files.output = $${OUT_PWD}/datafiles/${QMAKE_FILE_BASE}${QMAKE_FILE_EXT}
copy_files.commands = ${COPY_FILE} ${QMAKE_FILE_IN} ${QMAKE_FILE_OUT}
copy_files.CONFIG += no_link target_predeps

QMAKE_EXTRA_COMPILERS += copy_files

Add your big files to MYFILES variable. For each file a rule will be generated in Makefile that copies file to specified directory (datafiles in the example). Original file will be listed as a dependecy in the rule (this is default qmake behaviour) so copy will occur only when original file is fresher than existing copy. Generated rules are listed as dependencies in the target file rule (copy_files.CONFIG += target_predeps) so copying will occur on every build automatically.

The only caveat is this: if your set of files is dynamic (files are added or removed) you can use masks as in my example but you have to be careful to execute qmake after changing the set. Be aware that Qt Creator builds projects by launching make, not qmake. The most simple way to ensure that qmake will be launched is to modify .pro file.

For those who can read Russian there is more info about QMAKE_EXTRA_COMPILERS here

Answer 2

Do you need the script to be cross platform? I personally wouldn't use the copy command, but robocopy on Windows and rsync on Mac/Linux.

win32: $${DEMO_DATA_NAME}.commands = robocopy $${SRC_DIR} $${DST_DIR} $${FILE} /MIR /XO
!win32: $${DEMO_DATA_NAME}.commands = rsync -aru $${FILE} $${BLD_DATA_DIR}

I'm not really sure what you want to copy here, but you get the idea, you can adapt the files and/or directories.

Robocopy parameters are described here.

• /MIR Mirrors a directory tree
• /XO Excludes older files.

Rsync parameters are described here.

• -a Archive
• -r Recursive
• -u Update only when the source is newer

As a side note if you don't want to run this make install command, you can set this extra target as a dependency to the project that needs these files: theProjectNeedingThoseFiles.depends += DemoData.