How can I prevent a directory from being removed in rsync

Question

I am trying to make a script for backing up my home and other important directories to an external drive, while excluding certain directories and filetypes (--exclude). Excluded files need to be deleted on the backup as well, in case I add a filetype or directory to the exclude list (--delete-excluded).

Also, everything that is removed needs to be backed up as well, in case something goes horribly wrong (--backup --backup-dir=rsync/backup).

I am starting out with this:

#!/usr/bin/env bash

PATH_PWD="`pwd`"
PATH_HOME=~
SRC="/home/redsandro"
PATH_TARGET="/media/redsandro/MyBook 4TB/backup/`hostname`"
PATH_BIN="${SRC}/bin"
EXCL_FROM="${PATH_BIN}/rs-backup-external.sh.exclude.txt"
DATE="`date +%Y-%m-%d`"

OPTS="-ahl --update --del --delete-excluded --force --ignore-errors --progress --stats --exclude-from=$EXCL_FROM --log-file=~/rsync/rsync.$DATE.log"
OPTS="$OPTS --backup --backup-dir=rsync/backup/${DATE}"
#OPTS="$OPTS --dry-run"

echo Backing up $SRC to $PATH_TARGET...
echo  
sudo rsync $OPTS "$SRC/" "${PATH_TARGET}${SRC}"

Now, obviously, --del conflicts with --backup --backup-dir=rsync/backup. Can I exclude this directory from being deleted? Or is there perhaps an easier way to do this?

My goal is to have a script that I can run on all my machines, where a part syncs to a computer-specific directory, and another part (documents, pictures) sync to a global directory because they all need to be the same anyway.

Answer 1

Rsync is an awesome tool, but sometimes difficult to use.

The problem is you're using --exclude-from, which sets global exclude rules to both the sending and receiving side.

Before rsync performs operations on the filesystem, it will first assemble two lists of files, one for the sending side (client) and other for the receiving side (server). If a file is excluded from both sides, rsync will simply ignore it. If it is excluded only from the sending side and exists on the receiving side, rsync will delete it.

In your case, since your rules are excluding files from both sides, rsync will not delete them. You then use --delete-excluded to force deletion. But, like you said, this option conflicts with --backup --backup-dir=rsync/backup.

I suggest you do the following.

1. Convert $EXCL_FROM to a normal filter file, i.e., insert the minus sign before each line to make them exclude rules.
2. Remove --exclude-from=$EXCL_FROM and --delete-excluded from the rsync options.
3. Include -FF on the rsync options.

4. Include the following command before the call to rsync.

   cp "$EXCL_FROM" "$SRC/.rsync-filter"
   

In this way, your rules will only apply to the sending side, and files specified as exclude rules will be removed from the receiving side if found. This should not conflict with backup options.

Update

Essentially, the -FF option is making rsync process per-directory filter files named .rysnc-filter in both sides of the transfer. In your case it will exist only on the sending side, so its rules won't apply to the receiving side.

Actually, -FF equals to -F -F. The first F tells rsync to merge rules from .rsync-filter in the root of the source and destination directories, like I said before. The second F tells rsync to exclude them. You don't want .rsync-filter being transferred, otherwise your exclude rules would also apply to the receiving side and your original problem would return – the excluded files would cease to be deleted.

Here's what my man (1) rsync says:

-F    same as --filter='dir-merge /.rsync-filter'
      repeated: --filter='- .rsync-filter'