shell-scripting: Use a pipe as an input for tar

Question

I'm trying to figure out a way to use tar+pipes on a Ubuntu Server LTS.

I've got a postgresql command (pg_dump) that outputs lots of sql on the standard output:

pg_dump -U myUser myDB

I know how to redirect that to a file:

pg_dump -U myUser myDB > myDB.sql

In order to save some disk space, I would rather have it compressed: I can do a tar.gz file from that myDB.sql, and then delete myDB.sql.

But I was wondering - is there a way of doing this without creating the intermediate .sql file? I believe this could be accomplished with pipes... however I'm no shell guru, and know very little about them (I'm able to do ls | more, that's all). I've tried several variations of pg_dump .. | tar ... but with no success.

How can I use a pipe to use the output of pg_dump as an input for tar? Or did I just get something wrong?

Answer 1

In your use case pg_dump creates only a single file which needs to be compressed. As others have hinted, in *nix land an archive is a single file representing a filesystem. In keeping with the unix ideology of one tool per task, compression is separate task from archival. Since an archive is a file it can be compressed, as can any other file. Therefore, since you only need to compress a single file, tar is not necessary as others have already correctly pointed out.

However, your title and tags will bring future readers here who might be expecting the following...

---

Let's say you have a whole folder full of PostgreSQL backups to archive and compress. This should still be done entirely using tar, as its -z or --gzip flag invokes the gzip tool.

So let's also say you need to encrypt your database archives in preparation for moving them to a dubiously secured offsite backup solution (such as an S3-compatible object store). And let's assume you like pre-shared token (password) encryption using the AES cipher.

This would be a valid situation where you might wish to pipe data to and from tar.

Archive -> Compress -> Encrypt

tar cz folder_to_encrypt | openssl enc -aes-256-cbc -e > out.tar.gz.enc

Decrypt -> Uncompress -> Extract

openssl enc -aes-256-cbc -in ./out.tar.gz.enc -d | tar xz --null

Do refer to the GNU tar documentation for details of how the --null flag works and more useful examples for other situations where you might need to pipe files to tar.

Answer 2

I don't see how "tar" figures into this at all; why not just compress the dump file itself?

pg_dump -U myUser myDB | gzip > myDB.sql.gz

Then, to restore:

gzip -cd myDB.sql.gz | pg_restore ...

The "tar" utility is for bundling up a bunch of files and directories into a single file (the name is a contraction of "tape archive"). In that respect, a "tar" file is kind-of like a "zip" file, except that "zip" always implies compression while "tar" does not.

Note finally that "gzip" is not "zip." The "gzip" utility just compresses; it doesn't make archives.