How to perform successful SSL encryption with pkcs12/pfx in Qt on Mac OSX?

Question

Novice to Qt and developing a cross platform app, which requires SSL authentication from the server as well as client sides The .pem based encryption is working on Linux, Android, Windows. However there are problems with Mac OSX. Our code looks like below:

QFile privateKeyFile(":/Certificate.pem");  // --> has certificate + key
privateKeyFile.open(QIODevice::ReadOnly | QIODevice::Text);      
setLocalCertificateChain(QSslCertificate::fromPath(":/Certificate.pem", QSsl::Pem));
setPrivateKey(QSslKey(privateKeyFile.readAll(), QSsl::Rsa));

In above code privateKey().isNull() returns true for Mac. When we referred this post, it says that Mac doesn't support .pem based encryption.

The Secure Transport back-end to curl only supports client IDs that are in PKCS#12 (P12) format; it does not support client IDs in PEM format because Apple does not allow us to create a security identity from an identity file in PEM format without using a private API. And we can't use the private API, because apps that use private API are not allowed in any of Apple's app stores.

With my limited understanding, I interpreted that .pem is not a good idea for SSL communication with the server. Please stop me if it's wrong!

Hence, we decided to move to .pfx for all the platforms. We already had a .pfx file with a passphrase. We converted above code to be compatible with .pfx (i.e. "Certificate.pfx", we had this old file along with "Certificate.pem"). Instead of QSsl::Pem, we tried QSsl::Der. But as expected, it didn't work. However, there was no encryption error either, but we are sure that we are doing something wrong. :-)

We referred this post and try to regenerate a .pfx from .pem, but that also didn't help.
QSslCertificate::importPkcs12 fails to parse PFX file
In above case, the QSslCertificate::importPkcs12() returns false for the original .pfx file. Even if we generate a new .pfx from the command line, that also fails for the above function.

Question: Can someone help with exact way of performing the .pfx encryption with the server?
.pem authentication is also fine.

Note:

• Server supports both .pfx & .pem. That we confirmed with regular C OpenSSL libraries. But we want to achieve it using Qt.
• We are open to formats other than .pfx, should they work in all the platforms

Answer 1

DISCLAIMER: I am writing this from the top of my mind, since I don't personally own a Mac and cannot verify it anymore.

We had this exact problem about a year or two ago at my last job. It all boils down to Apple dropping support for OpenSSL.

Because of that, Qt switched from OpenSSL backend to Secure Transport backend on Mac with Qt5.6. Now the Secure Transport implementation is lacking some features. For example we were not able to load private key pem-files. I think switching from PKCS#8 to PKCS#1 helped, which can both be stored in .pem files and look almost identical, so that took a while to figure out.

We also noticed that a successfully loaded private key will be stored inside the Mac's key store and could be viewed and exported from there by the user, which we also did not want.

We finally went with re-compiling the QtNetwork module to use OpenSSL instead of Secure Transport. You will need to provide OpenSSL for that, since OSX does not include the headers anymore. A homebrew installation was sufficient I think. Other than that the compilation was surprisingly painless and fast, since you just have to compile one small module, not the whole Qt.

The easiest way to do this is:

1. download the source distribution of the Qt version you are running
2. ./configure it to use OpenSSL (the -openssl switch I believe)
3. cdinto the network folder
4. make
5. copy the generated QtNetwork.framework inside your Qt-Installation and replace the existing one.

With that everything worked as expected.