OpenSSL decrypt fails but error code is 0

Question

When I try to decrypt an encrypted S/MIME message using CMS in OpenSSL, the decrypt method returns me 0 which stands for didn't succeed.

OpenSSL.org says..

CMS_decrypt() returns either 1 for success or 0 for failure. The error can be obtained from ERR_get_error(3)

When I run this...

out = BIO_new(BIO_s_mem());
if (!out)
        assert(false);

int error = CMS_decrypt(cms, rkey, rcert, out, NULL, 0);
    if (!error) {
    fprintf(stderr, "Error Decrypting Data\n");
    printf("error code: %d\n", ERR_get_error());
    ERR_print_errors_fp(stderr);
    assert(false);
}

... the error variable is 0 which means an error occurred and the error code from ERR_get_error() is also 0. Additionally ERR_print_errors_fp() doesn't print anything which means there was no error.

The output from the aforementioned code:

Error Decrypting Data
error code: 0
Assertion failed: (false)

Does anyone have a suggestion what's going wrong here? Thanks

Answer 1

After many weeks of trying different approaches and frustration I ended up using PKCS#7 decrypt of OpenSSL. Due CMS is basically based on PKCS#7 it works fine with encryption using CMS and decryption using PKCS7.

The method I have used is pkcs7_decrypt().