rsync incremental file list taking forever

Question

I'm copying from one NAS to another. (Netgear ReadyNAS -> QNAP) i tried Pulling the files by running rsync on the QNAP, and that took forever, so I'm currently trying to push them from the Netgear. The code I'm using is:

rsync -avhr /sauce/folder [email protected]:/dest/folder

i'm seeing:

sending incremental file list

and nothing after that.

File transfer is 577gb and there are a lot of files, however I'm seeing 0 network traffic on the QNAP (It fluctuates between 0kb/s to 6kb/s) so it looks like its not sending any kind of incremental file list.

all folders are created on the destination and then nothing happens after that.

Anyone have any thoughts? Or any ideas on if there is a better way to copy files from a ReadyNAS to QNAP

Answer 1

The documentation for -v says increase verbosity. If the only thing you're interested in is seeing more progress, you can chain -v together like so:

rsync -avvvhr /sauce/folder/ [email protected]:/dest/folder/

and you should see more interesting progress.

This could tell you if your copying requirements -a are stricter than you need and thus take a lot of unnecessary processing time. For example, I attempted to use -a, which is equivalent to -rlptgoD, on over 100,000 images. Sending the incremental file list did not finish, even overnight. After changing it to

rsync -rtvvv /sauce/folder/ [email protected]:/dest/folder/

sending the incremental file list became much faster, being able to see file transfers within 15 minutes

Answer 2

After leaving it over night and it doing nothing, i came in and tried again.

the code that worked appended a '*' to the end of the sauce folder. so this was what worked:

rsync -avhr /sauce/folder/* [email protected]:/dest/folder

If anyone else has troubles - give this a shot.

Answer 3

My encounter with this was a large file that was incomplete but considered "finished transfer". I deleted the large (incomplete) file on the remote side and did another sync, which appears to have resolved the issue.

Answer 4

I am using QNAP 1 as production system and QNAP 2 as a backup server. On QNAP 1, I use the following script as cronjob to copy files in regular intervals to the backup-QNAP. Maybe you could try this:

DATUM=`date '+%Y-%m-%d'`;
MAILFILE="/tmp/rsync_svn.txt"
EMAIL="[email protected]"

echo "Subject: SVN Sync" > $MAILFILE
echo "From: $EMAIL" >> $MAILFILE
echo "To: $EMAIL" >> $MAILFILE
echo "" >> $MAILFILE
echo "-----------------------------------" >> $MAILFILE

   rsync -e ssh -av /share/MD0_DATA/subversion 192.168.2.201:/share/HDA_DATA/subversion_backup >> $MAILFILE 2>&1

echo "-----------------------------------" >> $MAILFILE
cat $MAILFILE | sendmail -t