In place encryption/decryption using EVP_*

Question

I am using openssl in my C++ project to encrypt and decrypt text files. As it has already been indicated here, I am following the tutorial from: https://wiki.openssl.org/index.php/EVP_Symmetric_Encryption_and_Decryption. I noticed that in this tutorial separate buffers are used for plain and cyphered texts. Since the data may occupy lots of memory and the algorithm is performed block-wise, I am wondering is it possible to perform in-place encryption/decryption, so that the output data is put to the same buffer as the input data, when it is no longer needed?

Answer 1

The OpenSSL 1.1.1 docs for EVP_EncryptUpdate() are here:

https://www.openssl.org/docs/man1.1.1/man3/EVP_CIPHER_CTX_new.html

This sentence at the end of the EVP_EncryptUpdate() paragraph talks about overlapping the input and output buffers:

It also checks if in and out are partially overlapping, and if they are 0 is returned to indicate failure.

Checking the actual source code we see this:

int is_partially_overlapping(const void *ptr1, const void *ptr2, int len)
{
    PTRDIFF_T diff = (PTRDIFF_T)ptr1-(PTRDIFF_T)ptr2;
    /*
     * Check for partially overlapping buffers. [Binary logical
     * operations are used instead of boolean to minimize number
     * of conditional branches.]
     */
    int overlapped = (len > 0) & (diff != 0) & ((diff < (PTRDIFF_T)len) |
                                                (diff > (0 - (PTRDIFF_T)len)));

    return overlapped;
}

...

if (is_partially_overlapping(out + ctx->buf_len, in, cmpl)) {
  EVPerr(EVP_F_EVP_ENCRYPTDECRYPTUPDATE, EVP_R_PARTIALLY_OVERLAPPING);
  return 0;
}

So fully overlapping input and output buffers are permitted (for non-custom ciphers). It's only partially overlapping that aren't allowed. For custom ciphers, the cipher itself is consulted for overlapping rules.

Answer 2

This is a general answer, not directed only toward EVP_*.

For some implementations yes but there are potential problems with encryption and padding.

For encryption the input data storage would need to be made increased to accept the padding but that would change the input data thus changing the encrypted data. In some implementations where the input length can be specified explicitly this can work.

For decryption with padding the data size will be smaller and this must be handles in some manner, again some implementations will allow this to be accomplished.

But unless the implementation explicitly states using the same data space is supported this would not be recommended. It may work for some input but not for all input and may break if the implementation internals are changed in the future.

Note: I have successfully tested this on one implementation (not EVP_*).