How do I copy with scp with a wildcard(*) in the destination path?

Question

Is it possible to get the value of what a wildcard matched to? I.e. I want to do

scp -r user@host:some/path/with/*/file.txt cp/to/*/file.txt

With the command above, I want

some/path/with/TEST/file.txt --> cp/to/TEST/file.txt
some/path/with/OTHER/file.txt --> cp/to/OTHER/file.txt

EDIT: I'll add some more data of my actual setup, which might make it clearer.

On my server I have the structure

server $ pwd
~/sixtrack/simulations
server $ ls
run0001
run0002
run0003
...

Each run* is a directory, containing files like run*/summary.dat, run*/tracks.dat etc. I could copy everything over to my own computer like

pingul $ scp -r 'user@host:sixtrack/simulations/*' .

However, I would like to not copy over every file, instead just some. Because all files are named the same and only separated through the directory, I have to respect the structure. As such, I would like host:sixtrack/simulations/run0001/summary.dat to go into my local folder /Users/pingul/Workspace/simulations/run0001/summary.dat.

Running

pingul $ pwd
/Users/pingul/Workspace/simulations
pingul $ scp -r 'user@host:sixtrack/simulations/*/summary.dat' */.

Does not accomplish what I want. Adding the -v option as suggested prints out

Executing: cp -r -- run0001/. run0100/.
Executing: cp -r -- run0002/. run0100/.
Executing: cp -r -- run0003/. run0100/.
...

and I only get one summary.dat in run0100. All other directories are empty (i.e. run0001, run0002, ...)

Is what I want possible?

Answer 1

It depends on whether you want to expand * before running a command or after running it and whether you want to do it interactively or not.

If you want to expand it before running a command interactively you can either use insert-completions (M-*) or glob-expand-word (C-x *) described in man bash:

   glob-expand-word (C-x *)
          The word before point is treated as a pattern for
          pathname expansion, and the list of matching filenames
          is inserted, replacing the word.  If a numeric argument
          is sup- plied, an asterisk is appended before pathname
          expansion.

   insert-completions (M-*)
          Insert all completions of the text before point that
          would have been generated by possible-completions.

To use these functions put a cursor before or after * and press either Control-x * or Alt-*:

$ pwd
/tmp/expand-glob
$ ls
FILE
$ cp /tmp/expand-*/*

Now put your cursor after the last *, don't press Enter but C-x * and you'll get this

$ cp /tmp/expand-glob/FILE

If you want to expand * to test command in the script then neither scp nor cp is a good option because the cannot run in dry-run mode. You should go with something like rsync that would show what files it would transfer if it was actually run like this:

$ rsync  -vn --relative /tmp/expand-*/* .
/tmp/
/tmp/expand-scp/
/tmp/expand-scp/a
/tmp/expand-scp/b
/tmp/expand-scp/c
/tmp/expand-scp/mmmm32

EDIT:

How about this:

$ rsync -avn -R --rsync-path="cd sixtrack/simulations && rsync"  user@host:run*/summary.dat .

-n stands for dry-run. With this option rsync will only print how will recreate a remote directory structure in current directory. In your case it will be something like:

receiving incremental file list
drwxr-xr-x          4,096 2016/10/13 12:45:36 run0001
-rw-r--r--              0 2016/10/13 12:23:18 run0001/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run00010
-rw-r--r--              0 2016/10/13 12:23:18 run00010/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0002
-rw-r--r--              0 2016/10/13 12:23:18 run0002/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0003
-rw-r--r--              0 2016/10/13 12:23:18 run0003/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0004
-rw-r--r--              0 2016/10/13 12:23:18 run0004/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0005
-rw-r--r--              0 2016/10/13 12:23:18 run0005/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0006
-rw-r--r--              0 2016/10/13 12:23:18 run0006/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0007
-rw-r--r--              0 2016/10/13 12:23:18 run0007/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0008
-rw-r--r--              0 2016/10/13 12:23:18 run0008/summary.dat
drwxr-xr-x          4,096 2016/10/13 12:23:18 run0009
-rw-r--r--              0 2016/10/13 12:23:18 run0009/summary.dat

Answer 2

What you're asking for is impossible since cp/scp commands take only a single destination. However you can easily simulate it with some very straightforward scripting:

for d in *
do
    scp -r user@host:some/path/with/"$d"/file.txt cp/to/"$d"/file.txt
done

It can also be written on a single line as

for d in *; do scp -r user@host:some/path/with/"$d"/file.txt cp/to/"$d"/file.txt; done