Mixing python and bash shell's xargs command doubts

Question

system : Centos 6.7 Lasted 
Shell : bash 
python : 2.6.6

It makes me very confused! example below:

5 files:

a1111  a2222  b1111  b2222  t.py

t.py content:

import sys

if __name__ == '__main__':
    a1 = sys.argv[1]
    print 'id(a1)=%s, len(a1)=%s, str(a1)=%s, type(a1)=%s' % (id(a1), len(a1), str(a1), type(a1))

do this like:

ls | xargs -I{} echo $(python t.py '{}')

output:

id(a1)=139821454683184, len(a1)=2, str(a1)=a1111, type(a1)=<type 'str'>
id(a1)=139821454683184, len(a1)=2, str(a1)=a2222, type(a1)=<type 'str'>
id(a1)=139821454683184, len(a1)=2, str(a1)=b1111, type(a1)=<type 'str'>
id(a1)=139821454683184, len(a1)=2, str(a1)=b2222, type(a1)=<type 'str'>
id(a1)=139821454683184, len(a1)=2, str(a1)=t.py, type(a1)=<type 'str'>

my question is why len(a1)=2, but str(a1)=a1111 ?， string length is obviously not equal to 2，

with no echo is ok, but it's not my question. I have use xargs -p option to print the cmd

ls | xargs -I{} python t.py '{}'

Answer 1

The reason this happens is that the $(python t.py '{}') expression is being evaluated before it gets passed to xargs. $(python t.py '{}') prints "id(a1)=139821454683184, len(a1)=2, str(a1)={}, type(a1)=", so that gets passed to xargs, which replaces the {} with each of the filenames...

Here's a shell trace that shows what's happening:

$ set -x # turn on tracing
$ ls | xargs -I{} echo $(python t.py '{}')
+ ls
++ python t.py '{}'
+ xargs '-I{}' echo 'id(a1)=4560222208,' 'len(a1)=2,' 'str(a1)={},' 'type(a1)=<type' ''\''str'\''>'
id(a1)=4560222208, len(a1)=2, str(a1)=a1111, type(a1)=<type 'str'>
id(a1)=4560222208, len(a1)=2, str(a1)=a2222, type(a1)=<type 'str'>
id(a1)=4560222208, len(a1)=2, str(a1)=b1111, type(a1)=<type 'str'>
id(a1)=4560222208, len(a1)=2, str(a1)=b2222, type(a1)=<type 'str'>
id(a1)=4560222208, len(a1)=2, str(a1)=t.py, type(a1)=<type 'str'>

The "+" lines show what the shell is actually executing. (You can ignore that the arguments to xargs are shown in single quotes; this is just because the output of t.py was split into words, but the other shell metacharacters in its output were ignored, and to get the same effect directly on the command line you'd have to quote (/escape) the arguments.)

BTW, there's another giveaway that this is what's happening: the id is the same on each line, but if t.py was actually being executed separately for each line, you'd get different ids.

Answer 2

When you are seeing this output:

len(a1)=2

This is actually length of replStr that is being used after -I option of xargs. This is {} hence length is 2.

If you use a different replStr.

ls | xargs -I% echo $(python ./t.py '%')

then you will see len=1 in output since length of % is 1.

id(a1)=4342949968, len(a1)=1, str(a1)=a1111, type(a1)=<type 'str'>
id(a1)=4342949968, len(a1)=1, str(a1)=a2222, type(a1)=<type 'str'>
id(a1)=4342949968, len(a1)=1, str(a1)=b1111, type(a1)=<type 'str'>
id(a1)=4342949968, len(a1)=1, str(a1)=b2222, type(a1)=<type 'str'>
id(a1)=4342949968, len(a1)=1, str(a1)=t.py, type(a1)=<type 'str'>

Reason of that is when you use:

ls | xargs -I{} echo $(python ./t.py '{}')

you are actually calling python program in a subshell and that is passing literal {} to python.

Here is the debug output to show the same:

bash -cx 'ls | xargs -I{} echo $(python ./t.py "{}")' >/dev/null
+ ls
++ python ./t.py '{}'
+ xargs '-I{}' echo 'id(a1)=4460329040,' 'len(a1)=2,' 'str(a1)={},' 'type(a1)=<type' ''\''str'\''>'

However of you use:

bash -cx 'ls | xargs -I{} python ./t.py "{}"' >/dev/null
+ ls
+ xargs '-I{}' python ./t.py '{}'

You can see different behavior in the way xargs is calling python command line.