Atomic rsync at directory level with minimum temporary storage

Question

I have some files on remote host (in a directory) and I want to perform rsync in an atomic manner at directory level to pull files on local host (In a distributed setup). One way I could think of is a very trivial case when I can take files backup on local host and then replace the old files with the new files, but the approach is not efficient as far as disk space is concerned. e.g. files size is 10GB and diff is just 100 MB.

Is there a way to store just the rsync diff on local host in temporary location and then update the files on local host?

Answer 1

You could do it like this:

1. Run rsync between local host and a temp folder in remote host. To make sure you only get the diff, use the --link-dest option and link to the real folder in remote host.

   You'd basically have a command like this:

   rsync --link-dest="/var/www" --archive "/localhost/path/www/" "[email protected]:/var/www_update_20131129"

   (With /var/www being the files to update and /var/www_update_20131129/ being the "temp" folder)

2. Once the rsync operation is done, you can swap the www_update_20131129/ and real www/ folders in remote host (possibly by soft-linking www/ to www_update_20131129/).