Rsync: Combining '--files-from' and '--exclude' does not work

Question

I do my backups with rsync and used the option --files-from to define a list of directories that should be synced. Right now, I am planning to extend the command by the --exclude option, but it does not work.

More specifically, the include-list.txt file already contains the directory /home/user/shared/ and I want to exclude the new subdirectory /home/user/shared/BIG_DATA/.

Here is the command that does not do the job.

rsync -azvvr \
    --files-from '/home/user/BACKUP_MNGMT/include-list.txt' \
    --exclude '/home/user/BACKUP_MNGMT/exclude-list.txt' \
    -e ssh / user@server:/home/user/Backup

Answer 1

You can combine --files-from and --exclude but it's not quite as straightforward as one might think. (TL;DR? Jump to the end.)

Consider a simple example. Here is the set-up

mkdir -p dst src/a src/b/c
n=1
for d in $(find src -type d)
do
    ps -ef > "$d"/file$n.txt
    n=$((n+1))
done

The result is this directory tree, which we'll use for our examples:

src/file1.txt
src/a/file2.txt
src/b/file3.txt
src/b/c/file4.txt

Using rsync in the trivial case we get everything, as expected:

rsync -av --dry-run src/ dst/
file1.txt
a/
a/file2.txt
b/
b/file3.txt
b/c/
b/c/file4.txt

Using rsync --exclude b you can see that we exclude b successfully. What's actually happening here is that because b is no longer considered, none of the files underneath b are found or copied:

rsync -av --dry-run --exclude b/ src/ dst/
file1.txt
a/
a/file2.txt

The problem is that when you provide a list of files that includes entries under b, the exclusion of b itself is applied but the files underneath it do not match the --exclude b clause and are therefore included:

find src | rsync -av --dry-run --files-from=- --exclude b/ . dst/
src/
src/file1.txt
src/a/
src/a/file2.txt
src/b/
src/b/file3.txt
src/b/c/
src/b/c/file4.txt

The solution is to specify in the --exclude clause that not only b but also all its descendants are to be excluded:

find src | rsync -av --dry-run --files-from=- --exclude b/ --exclude 'b/**' . dst/
src/
src/file1.txt
src/a/
src/a/file2.txt

---

In your case, you also have a number of other issues with your command. Here are a couple of variants that may work, depending on what is actually in the files lists:

rsync -azvv --no-r \
    --include-from '/home/user/BACKUP_MNGMT/include-list.txt' \
    --exclude-from '/home/user/BACKUP_MNGMT/exclude-list.txt' \
    -e ssh / user@server:/home/user/Backup

rsync -azvv \
    --files-from '/home/user/BACKUP_MNGMT/include-list.txt' \
    --exclude '/home/user/shared/BIG_DATA/**' \
    -e ssh / user@server:/home/user/Backup