Making GCM/CBC ciphers streamable in golang

Question

The GCM and CBC AES ciphers in Go can't be used along side with StreamWriter or StreamReader, which forces me to allocate the entire file into memory. Obviously, this is not ideal with large files.

I was thinking of making them streamable, by allocation some fixed size of blocks into memory, and feeding them to GCM or CBC, but I'm assuming that is probably a bad idea, since there must be a reason they've been designed this way.

Can someone explain why these operation modes can't be used without allocating the entire files into memory?

Answer 1

Simple answer - that's how they designed the API.

CBC and GCM are very different modes. GCM is AEAD mode (Authenticated Encryption with Associated Data). Do you actually need authentication? If not, for large files CBC is a good fit. You could also use CTR or OFB but they're streaming modes and are very strict about choosing an IV.

Before implementing anything I suggest you read about these modes. You have to at least understand, which parameters they need, for what purpose and how they should be generated.

CBC

BlockMode interface is a good fit for encrypting large files. You just need to encrypt block by block. CBC requires padding, but Go doesn't have the implementation. At least, I don't see one. You will have to deal with that somehow.

GCM

GCM uses AEAD interface, which only allows you to encrypt and decrypt a whole message. There is absolutely no reason why it should be implemented like that. GCM is a streaming mode, it's actually a good fit for streaming encryption. The only problem is authentication. GCM produces a tag at the end, which acts like a MAC. To make use of that tag you can't just encrypt an endless stream of data. You have to split it into chunks and authenticate them separately. Or do something else but at some point you have to read that tag and verify it, otherwise there's no point in using GCM.

What some libraries do, including Go, is they append that tag at the end implicitly on encryption and read and verify it on decryption. Personally, I think that's a very bad design. Tag should be available as a separate entity, you can't just assume that it will always be at the end. And that's not the only one of the problems in Go implementation. Sorry for that rant. Recently I've dealt with a particulary bad implementation.

As a solution I suggest you split your stream into chunks and encrypt them separately with a unique nonce (that's very important). Each chunk will have a tag at the end, which you should verify. That way you can make use of GCM authentication. Yes, it's kind of ugly but Go doesn't give you access to inner methods, so that you could make your own encryption API.

As an alternative you could find a different implementation. Maybe even a C library. I can suggest mbedtls. For me, it's the best implementation I came across in terms of API.

Answer 2

Heres a stream implementation for reading from a BlockMode. Your mileage may vary.

type BlockReader struct {
    buf   []byte
    block cipher.BlockMode
    in    io.Reader
}

func NewBlockReader(blockMode cipher.BlockMode,reader io.Reader) *BlockReader {
    return &BlockReader{
        block: blockMode,
        in:    reader,
    }
}

func (b *BlockReader) Read(p []byte) (n int, err error) {
    toRead := len(p)
    mul := toRead/b.block.BlockSize()
    size := mul*b.block.BlockSize()
    if cap(b.buf) != size{
        b.buf = make([]byte,toRead,toRead)
    }

    read, err := b.in.Read(b.buf)
    if err != nil {
        return 0,err
    }

    if read < b.block.BlockSize(){
        return 0,io.ErrUnexpectedEOF
    }
    b.block.CryptBlocks(b.buf,b.buf)
    return copy(p,b.buf),nil
}