Programmatically verify a X509 certificate and private key match

Question

I created an RSA key pair using the EVP_aes_256_cbc() cipher. The private key is PEM encoded and has a passphrase. This requires a passphrase to be entered by the user.

Here's the create private key call:

//Save private key
    bio_priv = BIO_new_file(full_asymKeyFilePath.c_str(), "a+");
    if (PEM_write_bio_RSAPrivateKey(
        bio_priv,   //BIO handle
        rsa,        //Key handle
        EVP_aes_256_cbc(),      //Cipher encoding format
        pwd,        //Password
        pwd_len,            //Password length
        NULL,       //Callback
        NULL        //Not sure
        ) != 1) {
            //report err
    } 

Then I generated a certificate and signed it with the private key.

//Sign the certificate with the generated key
    if (!X509_sign(cert, evpKey, EVP_sha1())){
        //report err
    }

Later on, I want to verify that this certificate matches this RSA key pair. When I SSL_CTX_check_private_key(), I'm prompted to input a passphrase from the console.

Is there are way to automatically input the password so that I don't get prompted from the console?

//Load server certificate, must be called before ever calling use private key
    if (SSL_CTX_use_certificate_file(context, full_certFilePath.c_str(), SSL_FILETYPE_PEM) == 0){   //load all certs from PEM file into SSL_CTX 
        //err
    }

    //Load private key corresponding to the certificate
    if (SSL_CTX_use_PrivateKey_file(context, full_asymKeyFilePath.c_str(), SSL_FILETYPE_PEM) == 0){ //load all certs from PEM file into SSL_CTX 
        //file type is not pem or private key was loaded before calling this function. Private key does not match the public key in the certificate
        //err
    }

    //Verify that certificate and private key match
    if (!SSL_CTX_check_private_key(context)){  //<====== Prompts me to enter pass :(
        //err
    }

Answer 1

Programmatically verify a X509 certificate and private key match. Private key has a PEM passphrase

There are two answers here. One is for the certificate, and the second is for the private key. The private key is shown first because it is used to validate the certificate (so it makes sense to visit it first).

Also, its important to call the *_check_key routines because OpenSSL only checks that a key is well encoded; and it does not check that its actually valid. See, for example, Private key generated by openssl does not satisfy n = p * q.

---

In OpenSSL, you would use the following to verify the the private key is well encoded:

FILE* file = fopen(...);
EVP_PKEY* pkey = PEM_read_PrivateKey(file, NULL, PasswordCallback, NULL);
unsigned long err = ERR_get_error();

if(pkey)
    EVP_PKEY_free(pkey);

If pkey is NULL, then there was a problem and err holds a reason code. Otherwise, you have a properly encoded private key (but not necessarily valid).

If the key is properly encoded, you can check the type of private key and validate it with the following.

int type = EVP_PKEY_get_type(pkey);
switch (type)
{
case EVP_PKEY_RSA:
case EVP_PKEY_RSA2:
    RSA* rsa = EVP_PKEY_get1_RSA(pkey);
    rc = RSA_check_key(rsa);
    ASSERT(rc);
    RSA_free(rsa);

    break;

case EVP_PKEY_DSA:
case EVP_PKEY_DSA1:
case EVP_PKEY_DSA2:
case EVP_PKEY_DSA3:
case EVP_PKEY_DSA4:
    DSA* dsa = EVP_PKEY_get1_DSA(pkey);
    rc = DSA_check_key(dsa);
    ASSERT(rc);
    DSA_free(dsa);

    break;

case EVP_PKEY_DH:
    DH* dh = EVP_PKEY_get1_DH(pkey);
    rc = DH_check_key(dh);
    ASSERT(rc);
    DH_free(dh);

    break;

case EVP_PKEY_EC:
    EC_KEY* ec = EVP_PKEY_get1_EC_KEY(pkey);
    rc = EC_KEY_check_key(ec);
    ASSERT(rc);
    EC_KEY_free(ec);

    break;

default:
    ASSERT(0);
}

EVP_PKEY_get_type is not part of OpenSSL. Here's how I implemented it:

int EVP_PKEY_get_type(EVP_PKEY *pkey)
{
    ASSERT(pkey);
    if (!pkey)
        return NID_undef;

    return EVP_PKEY_type(pkey->type);
}

---

In OpenSSL, you would use the following to verify the the certificate is well encoded:

FILE* file = fopen(...);
X509* x509 = PEM_read_X509(file, NULL, NULL, NULL);
unsigned long err = ERR_get_error();

If x509 is NULL, then there was a problem and err holds a reason code. Otherwise, you have a properly encoded certificate (but not necessarily valid).

You can then verify the certificate with:

/* See above on validating the private key */
EVP_PKEY* pkey = ReadPrivateKey(...);

int rc = X509_verify(x509, pkey);
err = ERR_get_error();

If rc != 1, then there was a problem and err holds a reason code. Otherwise, you have a valid certificate and private key pair. If the certificate is valid, then you can't use err because err is only valid if there's a problem.

If your certificate is signed by an issuer (for example, a CA or intermediate), then you need to use a X509_STORE to verify the issuer's signature on your certificate (a lot of error checking omitted):

const char* serverCertFilename = ...;
const char* issuerCertFilename = ...;    

X509_STORE* store = X509_STORE_new();
ASSERT(store);

static const long flags = X509_V_FLAG_X509_STRICT | X509_V_FLAG_CHECK_SS_SIGNATURE
        | X509_V_FLAG_POLICY_CHECK;
rc = X509_STORE_set_flags(store, flags);
err = ERR_get_error();
ASSERT(rc);

/* Some other object/functions owns 'lookup', but I'm not sure which (perhaps the store) */
X509_LOOKUP* lookup = X509_STORE_add_lookup(store, X509_LOOKUP_file());
/* err = ERR_get_error(); // Does not set error codes. */
ASSERT(lookup);    

/* Cannot load this from memory. No API!!! */
rc = X509_LOOKUP_load_file(lookup, issuerCertFilename, X509_FILETYPE_PEM);
/* err = ERR_get_error(); // Does not set error codes. */
ASSERT(rc);

X509_STORE_CTX* ctx = X509_STORE_CTX_new();
ASSERT(ctx);

X509* serverCert = ReadCertifcate(serverCertFilename);
ASSERT(serverCert);

rc = X509_STORE_CTX_init(ctx, store, serverCert, NULL);
ret = err = ERR_get_error();
ASSERT(rc);

/* Error codes at https://www.openssl.org/docs/crypto/X509_STORE_CTX_get_error.html */
rc = X509_verify_cert(ctx);
err = X509_STORE_CTX_get_error(ctx);

/* Do cleanup, return success/failure */

---

Is there are way to automatically input the password so that I don't get prompted from the console?

Yes. use the password callback in PEM_read_PrivateKey. The PasswordCallback can simply provide a password in the buffer, or it can prompt the user and return the password in the buffer.

My password callback is somewhat involved. It performs a single hash of the raw password before passing it on to the library. That ensures a "plain text" password is not used (but does not slow down the customary attacks). Yours can prompt the user for a string, or it can return a hard coded string.

My password callback uses a label. The label allows me to derive different keys depending on usage (even though the same 'base' secret is used). By specifying a different usage or label, I get a different derivation of key bits. The label is provided through arg below, and you can set it with SSL_CTX_set_default_passwd_cb_userdata.

using EVP_MD_CTX_ptr = std::unique_ptr<EVP_MD_CTX, decltype(&::EVP_MD_CTX_destroy)>;

int PasswordCallback(char *buffer, int size, int rwflag, void *arg)
{
    UNUSED(rwflag);

    int rc;
    unsigned long err;
    ostringstream oss;

    const char* label = (char*) arg;
    size_t lsize = (label ? strlen(label) : 0);

    SecureVector sv = config.GetMasterKey();
    ASSERT(!sv.empty());
    if (sv.empty())
    {
        ...
        throw runtime_error(oss.str().c_str());
    }

    EVP_MD_CTX_ptr ctx(EVP_MD_CTX_create(), ::EVP_MD_CTX_destroy);
    ASSERT(ctx.get() != NULL);

    const EVP_MD* hash = EVP_sha512();
    ASSERT(hash != NULL);

    rc = EVP_DigestInit_ex(ctx.get(), hash, NULL);
    err = ERR_get_error();

    ASSERT(rc == 1);
    if (rc != 1)
    {
        ...
        throw runtime_error(oss.str().c_str());
    }

    rc = EVP_DigestUpdate(ctx.get(), sv.data(), sv.size());
    err = ERR_get_error();

    ASSERT(rc == 1);
    if (rc != 1)
    {
        ...
        throw runtime_error(oss.str().c_str());
    }

    if (label && lsize)
    {
        rc = EVP_DigestUpdate(ctx.get(), label, lsize);
        err = ERR_get_error();

        ASSERT(rc == 1);
        if (rc != 1)
        {
            ...
            throw runtime_error(oss.str().c_str());
        }
    }

    int n = std::min(size, EVP_MD_size(hash));
    if (n <= 0)
        return 0;

    rc = EVP_DigestFinal_ex(ctx.get(), (unsigned char*) buffer, (unsigned int*) &n);
    err = ERR_get_error();

    ASSERT(rc == 1);
    if (rc != 1)
    {
        ...
        throw runtime_error(oss.str().c_str());
    }

    return n;
}