How to encrypt files with AES256-GCM in golang?

Question

AES256-GCM could be implemented in go as https://gist.github.com/cannium/c167a19030f2a3c6adbb5a5174bea3ff

However, Seal method of interface cipher.AEAD has signature:

Seal(dst, nonce, plaintext, additionalData []byte) []byte

So for very large files, one must read all file contents into memory, which is unacceptable.

A possible way is to implement Reader/Writer interfaces on Seal and Open, but shouldn't that be solved by those block cipher "modes" of AEAD? So I wonder if this is a design mistake of golang cipher lib, or I missed something important with GCM?

Answer 1

AEADs should not be used to encrypt large amounts of data in one go. The API is designed to discourage this.

Encrypting large amounts of data in a single operation means that a) either all the data has to be held in memory or b) the API has to operate in a streaming fashion, by returning unauthenticated plaintext.

Returning unauthenticated data is dangerous it's not hard to find people on the internet suggesting things like gpg -d your_archive.tgz.gpg | tar xzbecause the gpg command also provides a streaming interface.

With constructions like AES-GCM it's, of course, very easy to manipulate the plaintext at will if the application doesn't authenticate it before processing. Even if the application is careful not to "release" plaintext to the UI until the authenticity has been established, a streaming design exposes more program attack surface.

By normalising large ciphertexts and thus streaming APIs, the next protocol that comes along is more likely to use them without realising the issues and thus the problem persists.

Preferably, plaintext inputs would be chunked into reasonably large parts (say 16KiB) and encrypted separately. The chunks only need to be large enough that the overhead from the additional authenticators is negligible. With such a design, large messages can be incrementally processed without having to deal with unauthenticated plaintext, and AEAD APIs can be safer. (Not to mention that larger messages can be processed since AES-GCM, for one, has a 64GiB limit for a single plaintext.)

Some thought is needed to ensure that the chunks are in the correct order, i.e. by counting nonces, that the first chunk should be first, i.e. by starting the nonce at zero, and that the last chunk should be last, i.e. by appending an empty, terminator chunk with special additional data. But that's not hard.

For an example, see the chunking used in miniLock.

Even with such a design it's still the case that an attacker can cause the message to be detectably truncated. If you want to aim higher, an all-or-nothing transform can be used, although that requires two passes over the input and isn't always viable.