Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

rsync skip non existing files on source

I'm developing a script in python that could collect logs from multiple machines. I'm using rsync. But there's a problem. I have logs for multiple services that look like:

service1.log
service2.log
service3.log
... and so on

Paths to this files and folders are specified in code. But sometimes I get in situation when some log files do not exist yet. And rsync does not finish successfully.
How can I skip files that do not exist on source machine?
P.S. I'm using saltstack to rule machines, so I call:

__salt__['cmd.retcode']('rsync -a file1 file2...')
like image 483
oleg.foreigner Avatar asked Dec 24 '14 11:12

oleg.foreigner


People also ask

Will rsync Skip existing files?

Rsync with --ignore-existing-files: We can also skip the already existing files on the destination. This can generally be used when we are performing backups using the –link-dest option, while continuing a backup run that got interrupted. So any files that do not exist on the destination will be copied over.

Does rsync check for existing files?

It checks to see if files exist in the destination before sending them, saving bandwidth and time for everything it skips. Also, rsync provides the ability to synchronize a directory structure (or even a single file) with another destination, local or remote.

Will rsync overwrite files?

Any that have been updated will be copied over, although note that rsync is extremely efficient in that only the changed parts of files are copied and if the file is exactly the same if it is not copied over at all. Any that have been deleted on the local system are deleted on the remote.

Why is rsync skipping files?

Conclusion. In short, this rsync error happens due to missing symlinks or lack of a recursive mode option while transferring data between servers.


1 Answers

Use --ignore-missing-args:

The version 3.1.0 man page says,

--ignore-missing-args

 When rsync is first processing the explicitly requested
 source files (e.g. command-line arguments or --files-from
 entries), it is normally an error if the file cannot be
 found.  This option suppresses that error, and does not
 try to transfer the file.  This does not affect subsequent
 vanished-file errors if a file was initially found to be
 present and later is no longer there.

For example:

% rsync --version
rsync  version 3.1.0  protocol version 31
% rsync bogusfile dest
rsync: link_stat "/tmp/bogusfile" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]
% rsync --ignore-missing-args bogusfile dest
# <no error>

The --ignore-missing-args option was added some time between version 3.06 and 3.1.0.

The online version 3.06 rsync man page does not mention it. But the git repository indicates this option was added in 2009.

like image 124
unutbu Avatar answered Oct 01 '22 02:10

unutbu