Install data directory tree with massive number of files using automake

Question

I have a data directory which I would like automake to generate install and uninstall targets for. Essentially, I just want to copy this directory verbatim to the DATA directory, Normally, I might list all the files individually, like

dist_whatever_DATA=dir/subdir/filea ...

But the problem arises when my directory structure looks like this

*root
 *subdir
  *~10 files
 *subdir
  *~10 files
 *subdir
  *~700 files
 *subdir
 ...
 ~20 subdirs

I just cannot list all 1000+ files included as part of my Makefile.am. That would be ridiculous.

I need to preserve the directory structure as well. I should note that this data is not generated at all by the build process, and is actually largely short audio recordings. So it's not like I would want automake to "check" that every file I want to install has actually been created, as they're either there or not, and whatever file is there, I know I want it to be installed, and whatever file is not, should not be installed. I know that this is the justification used in other places to not do wildcard instsalls, but all the possible reasons don't apply here.

Answer 1

I would use a script to generate a Makefile fragment that lists all the files:

echo 'subdir_files =' > subfiles.mk
find subdir -type f -print | sed 's/^/  /;$q;s/$/ \\/' >> subfiles.mk

and then include this subfiles.mk from your main Makefile.am:

include $(srcdir)/subfiles.mk
nobase_dist_pkgdata_DATA = $(subdir_files)

---

A second option is to EXTRA_DIST = subdir, and then to write custom install-data-local and uninstall-local rules.

The problem here is that EXTRA_DIST = subdir will distributes all files in subdir/, including backup files, configuration files (e.g. from your VCS), and other things you would not want to distribute.

Using a script as above let you filter the files you really want to distribute.

Answer 2

I've found that installing hundreds of files separately makes for a tormentingly long invocation of make install. I had a similar case where I wanted to install hundreds of files, preserving the directory structure, and I did not want to change my Makefile.am every time a file was added to or removed from the collection.

I included a LZMA archive of the files in my distribution, and made automake rules like so:

GIANTARCHIVE = My_big_archive.tar.lz

dist_pkgdata_DATA = $(GIANTARCHIVE)

install-data-hook:
    cd $(DESTDIR)$(pkgdatadir); \
    cat $(GIANTARCHIVE) | unlzma | $(TAR) --list > uninstall_manifest.txt; \
    cat $(GIANTARCHIVE) | unlzma | $(TAR) --no-same-owner --extract; \
    rm --force $(GIANTARCHIVE); \
    cat uninstall_manifest.txt | sed --expression='s/^\|$$/"/g' | xargs chmod a=rX,u+w

uninstall-local:
    cd $(DESTDIR)$(pkgdatadir); \
    cat uninstall_manifest.txt | sed --expression='s/ /\\ /g' | xargs rm --force; \
    rm --force uninstall_manifest.txt

This way, automake installs My_big_archive.tar.lz in the $(pkgdata) directory, and extracts it there, making a list of all the files that were in it, so it can uninstall them later. This also runs much faster than listing each file as an install target, even if you were to autogenerate that list.