I'm trying to write a parallel compress / encrypt backup script for archiving using GNU parallel, xz and GnuPG. The core part's of script is:
tar --create --format=posix --preserve-permissions --same-owner --directory $BASE/$name --to-stdout . \
| parallel --pipe --recend '' --keep-order --block-size 128M "xz -9 --check=sha256 | gpg --encrypt --recipient $RECIPIENT" \
| pv > $TARGET/$FILENAME
Without GnuPG encryption, it works great (uncompress and untar works), but after adding parallel encryption, it's fail to decrypt with below error:
[don't know]: invalid packet (ctb=0a)
gpg: WARNING: encrypted message has been manipulated!
gpg: decrypt_message failed: Unexpected error
: Truncated tar archive
tar: Error exit delayed from previous errors.
Because uncompressed size is as same as gnu parallel's block size(around 125M), I assume that it's related GnuPG's support of partial block encryption. How can I solve this problem?
FYI
Another parallel gpg encrption issue about random number generation
https://unix.stackexchange.com/questions/105059/parallel-pausing-and-resuming
Pack
tar --create --format=posix --preserve-permissions --same-owner --directory $BASE/$name --to-stdout . |
parallel --pipe --recend '' --keep-order --block-size 128M "xz -9 --check=sha256 | gpg --encrypt --recipient $RECIPIENT;echo bLoCk EnD" |
pv > $TARGET/$FILENAME
Unpack
cat $TARGET/$FILENAME |
parallel --pipe --recend 'bLoCk EnD\n' -N1 --keep-order --rrs 'gpg --decrypt | xz -d' |
tar tv
-N1
is needed to make sure we pass a single record at a time. GnuPG does not support decrypting multiple merged records.
GnuPG does not support concatenating multiple encryption streams and decrypting them at once. You will have to store multiple files, and decrypt them individually. If I'm not mistaken, your command even mixes up the outputs of all parallel instances of GnuPG, so the result is more or less random garbage.
Anyway: GnuPG also takes care of compression, have a look at the --compression-algo
option. If you prefer to use xz
, apply --compression-algo none
so GnuPG does not try to compress the already-compressed message again. Encryption has massive support by CPU-instructions ourdays, xz -9
might in fact be more time intensive than encryption (although I did not benchmark this).
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With