Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to `scp` directory preserving structure but only pick certain files?

Tags:

bash

scp

I need to secure copy (scp) to remotely copy a directory with its sub structure preserved from the UNIX command line. The sub directories have identically named files that I WANT and bunch of other stuff that I don't. Here is how the structure looks like.

directorytocopy
  subdir1
    1.wanted
    2.wanted
    ...
    1.unwanted
    2.notwanted
  subdir2
    1.wanted
    2.wanted
    ...
    1.unwanted
    2.notwanted
  ..

I just want the .wanted files preserving the directory structure. I realize that it is possible to write a shell (I am using bash) script to do this. Is it possible to do this in a less brute force way? I cannot copy the whole thing and delete the unwanted files because I do not have enough space.

like image 640
QuantSolo Avatar asked Jun 12 '15 15:06

QuantSolo


People also ask

Which option for SCP allows you to copy a directory and all its sub directories?

To copy a directory (and all the files it contains), use scp with the -r option. This tells scp to recursively copy the source directory and its contents.

Does SCP overwrite files by default?

By default, existing files are overwritten. To control overwrite behavior, use --overwrite. (If the files are identical no transfer occurs regardless of this setting value.) Because scp uses authentication and encryption provided by ssh, a Secure Shell server must be running on the remote computer.


2 Answers

Adrian has the best idea to use rsync. You can also use tar to bundle the wanted files:

cd directorytocopy
shopt -s nullglob globstar
tar -cf - **/*.wanted | ssh destination 'cd dirToPaste && tar -xvf -'

Here, using tar's -f option with the filename - to use stdin/stdout as the archive file.

This is untested, and may fail because the archive may not contain the actual subdirectories that hold the "wanted" files.

like image 134
glenn jackman Avatar answered Sep 24 '22 00:09

glenn jackman


Assuming GNU tar on the source machine, and assuming that filenames of the wanted files won't contain newlines and they are short enough to fit the tar headers:

find /some/directory -type f -name '*.wanted' | \
    tar cf - --files-from - | \
    ssh user@host 'cd /some/other/dir && tar xvpf -'
like image 43
lcd047 Avatar answered Sep 25 '22 00:09

lcd047