Can tar append b.tar onto a.tar where a.tar is taken from stdin?

Question

This is part of a git-archive-all style script which will generate a single tarball from a git project and all its submodules.

The submodule archives must be generated separately and appended to the superproject's archive. But I'd like to append them to the superproject archive on the fly, something like so:

git archive --format=tar ... > super.tar
# pseudo git foreach submodule ; do
  (cd $submodule && git archive --format=tar ...) | tar -Af super.tar
# done

But super.tar is unmodified. And tar -Af super.tar - is invalid.

I can just do it in two stages, but I'm hoping I'm just being stupid not seeing how to work this.

Below it's done in two steps:

git archive --format=tar ... > super.tar
# pseudo git foreach submodule ; do
  (cd $submodule && git archive --format=tar ...) > $submodule.tar
  tar -Af super.tar $submodule.tar && rm -f $submodule.tar
# done

Answer 1

tar is surprisingly finicky about piping for some reason! I tried various alternatives (see below) and checked all options in the manual and concluded that it simply is not possible with GNU tar 1.34.

Having determined that, there are alternatives, e.g., bsdtar, which works excellently with pipes and stdin/stdout. The syntax is a bit different though. You can use --append, which by default would add the whole archive into the existing archive. To only add the files inside the given archive to the result archive, there is a special @ prefix to be used, refer to the manual. This example works for me:

sudo apt install libarchive-tools  # package was named bsdtar previously
bsdtar --version
# bsdtar 3.4.3 - libarchive 3.4.3 zlib/1.2.11 liblzma/5.2.5
# bz2lib/1.0.8 liblz4/1.9.3 libzstd/1.4.8
echo foo > bar
# --no-fflags is to avoid the harmless "unknown extended header
# keyword 'SCHILY.fflags'" warning when extracting with GNU tar.
# I also force it to produce ustar because that is what GNU tar
# creates by default afaik.
bsdtar --format=ustar --no-fflags --create --file result.tar bar
tar tvlf result.tar
# -rwx------ user/group   4 2022-04-04 23:29 bar
echo bar > foo
echo argh > oof

With all that set up above to get a working example, this is the main command line you want:

tar -cf /dev/stdout foo oof |
    bsdtar --format=ustar --no-fflags --append --file result.tar @-
tar tvlf result.tar
# -rwx------ user/group   4 2022-04-04 23:29 bar
# -rwx------ user/group   4 2022-04-04 23:40 foo
# -rwx------ user/group   5 2022-04-04 23:40 oof

---

Here are the dead paths I tried out, they might show some alternatives if direct piping does not work and - is not accepted.

Instead of using - as input, you could try process substitution with <( ... ). Because I didn't want to use git archive, I had to force tar to compress to stdout to create some alternative input to be appended on-the-fly. But, using --file - only gives:

tar: Refusing to write archive contents to terminal

So, I had to trick tar to use --file /dev/stdout. Unfortunately, even this:

tar -Af super.tar <( tar -cf /dev/stdout foo )

prints no warning and simply leaves super.tar untouched. Reading from stdin instead of using a FIFO file had the same result:

git archive | tar -Af super.tar /dev/stdin