Transferring binary data over ADB shell (ie fast file transfer using tar) [duplicate]

Question

I've been trying to move files from my android device to my osx machine using something similar to: adb shell tar -c directory_to_copy | tar -x. It seems that the remote tar is working but somewhere on the way the file gets corrupted. After some playing around I've found:

• It seems the adb shell command translates LF to CRLF:

  % adb shell 'cd /mnt/sdcard;echo hi>a.bin'
  % adb shell 'cd /mnt/sdcard;cat a.bin' | hexdump -C
  00000000  68 69 0d 0a                                       |hi..|
  00000004
  % adb pull /mnt/sdcard/a.bin
  0 KB/s (3 bytes in 0.457s)
  % hexdump -C a.bin
  00000000  68 69 0a                                          |hi.|
  00000003
  

• It looks either the server or the daemon are causing that and not the client (see the len=4):

  % ADB_TRACE=1 adb shell 'cd /mnt/sdcard;cat a.bin'
  [... snip ...]
  system/core/adb/commandline.c::read_and_dump():read_and_dump(): post adb_read(fd=3): len=4
  [... snip ...]
  

I would guess that the daemon is doing that sort of translation in the shell command for windows users.

My questions are:

1. wtf? (what does that and for what purpose?)
2. is there any way to tell it (adbd?) to not do that?
3. can anyone think of any creative way to circumvent that (I thought about base64 encoding the data, but I would prefer to avoid that overhead. Also, creating a local file is not an option since my filesystem is quite full)

Thanks!

Answer 1

Adb isn't doing this on purpose, but it's not smart enough to refrain from allocating a terminal when you give it a command to run; the terminal on the android side is what's cooking the bytes. Instead of

adb shell 'cd /mnt/sdcard;cat a.bin' | hexdump -C

try doing

adb shell 'stty raw; cd /mnt/sdcard;cat a.bin' | hexdump -C

This instructs the terminal device not to mangle the bytes at all, but to pass them through.

Answer 2

Use adb exec-out <command> instead of adb shell.

Example adb exec-out cat /data/myfile.txt > localfile.txt

Answer 3

The "stty" trick mentioned in the first answer does not work in general.

For piping the output to the hexdump command on the host side it might be ok. However, for tar (as is mentioned in the subject of the original question) and for many other commands that can accept binary stream input, this does not work. As Fabian Zeindl has pointed out correctly, using 'adb exec-out ...' is the right way to go.

Some examples for transferring the output of tar over ADB:

1. Here it is assumed that the tar executable is found in your path on Android:

   • adb exec-out 'cd /sdcard; tar -cf - DCIM/' > DCIM.tar
   • adb exec-out 'cd /sdcard; tar -cf - DCIM/' | tar -tvf -

2. Some more complex examples involving compression and the use of busybox:

   • adb exec-out 'cd /sdcard; /system/xbin/extras/busybox tar -czf - DCIM/' > DCIM.tgz
   • adb exec-out 'cd /sdcard; GZIP="-9" /system/xbin/extras/busybox tar -czf - DCIM/' | tar -tvzf -
   • adb exec-out 'cd /sdcard; BGZIP2="-9" /system/xbin/extras/busybox tar -cjf - DCIM/' > DCIM.tar.bz2