I'm developing a script in python that could collect logs from multiple machines. I'm using rsync. But there's a problem. I have logs for multiple services that look like:
service1.log
service2.log
service3.log
... and so on
Paths to this files and folders are specified in code. But sometimes I get in situation when some log files do not exist yet. And rsync does not finish successfully.
How can I skip files that do not exist on source machine?
P.S. I'm using saltstack to rule machines, so I call:
__salt__['cmd.retcode']('rsync -a file1 file2...')
Rsync with --ignore-existing-files: We can also skip the already existing files on the destination. This can generally be used when we are performing backups using the –link-dest option, while continuing a backup run that got interrupted. So any files that do not exist on the destination will be copied over.
It checks to see if files exist in the destination before sending them, saving bandwidth and time for everything it skips. Also, rsync provides the ability to synchronize a directory structure (or even a single file) with another destination, local or remote.
Any that have been updated will be copied over, although note that rsync is extremely efficient in that only the changed parts of files are copied and if the file is exactly the same if it is not copied over at all. Any that have been deleted on the local system are deleted on the remote.
Conclusion. In short, this rsync error happens due to missing symlinks or lack of a recursive mode option while transferring data between servers.
Use --ignore-missing-args
:
The version 3.1.0 man page says,
--ignore-missing-args
When rsync is first processing the explicitly requested source files (e.g. command-line arguments or --files-from entries), it is normally an error if the file cannot be found. This option suppresses that error, and does not try to transfer the file. This does not affect subsequent vanished-file errors if a file was initially found to be present and later is no longer there.
For example:
% rsync --version
rsync version 3.1.0 protocol version 31
% rsync bogusfile dest
rsync: link_stat "/tmp/bogusfile" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]
% rsync --ignore-missing-args bogusfile dest
# <no error>
The --ignore-missing-args
option was added some time between version 3.06 and 3.1.0.
The online version 3.06 rsync man page does not mention it. But the git repository indicates this option was added in 2009.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With