ServiceFabric standalone: Failed to get private key file

Question

I have a standalone ServiceFabric cluster (3 nodes). I created SSL certificate for server and client authorization. Then I assign certificate thumbprint to a cluster config. Everything work okey( cluster health is Ok and my applications works as well. But there are a lot of errors in Microsoft-ServiceFabric/Admin log. Following warning and errors are writing to log every minute:

• CryptAcquireCertificatePrivateKey failed. Error:0x80090014
• Can't get private key filename for certificate. Error: 0x80090014
• All tries to get private key filename failed.
• Failed to get the Certificate's private key. Thumbprint: {Cert Thumbprint}. Error: E_FAIL
• Failed to get private key file. x509FindValue: {Cert Thumbprint}, x509StoreName: My, findType: FindByThumbprint, Error E_FAIL
• SetCertificateAcls failed. ErrorCode: E_FAIL Can't ACL
• FabricNode/ServerAuthX509FindValue, ErrorCode E_FAIL

I assinged write permitions to private keys storage for NETWORK SERVICE and SYSTEM. As well I assigned gMSA account for PK storage. But errors still apears in log. From the other hand everything looks fine, cluster up and running... Here is my cluster config (security part):

"security":{
"ServerCredentialType":"X509", "ClusterCredentialType":"Windows", "WindowsIdentities":{
"ClustergMSAIdentity":"[email protected]", "ClusterSPN":"http/servicefabric" }, "CertificateInformation":{
"ServerCertificate": { "Thumbprint": "{Cert Thumbprint}", "X509StoreName": "My" }, "ClientCertificateThumbprints":[
{
"CertificateThumbprint":"{Cert Thumbprint}", "IsAdmin":true } ], "X509StoreName": "My" } },

For x509 certificated creation I used OpenSSL 1.0.2k-fips 26 Jan 2017. I follow the steps from this article: https://gist.github.com/harishanchu/e82d759c0235379d1778f799992b5774 Could anyone clarify this issue?

Answer 1

It seems like you don't have a private key file in the MachineKeys folder. To verify if you have a physical file in the folder run this powershell command:

$certThumb = "1D6523F622E33DF46382D081BCA9AE9A2D8D78CC"

Try
{
     $WorkingCert = Get-ChildItem CERT:\LocalMachine\My |where {$_.Thumbprint -match $certThumb} | sort $_.NotAfter -Descending | select -first 1 -erroraction STOP
     $TPrint = $WorkingCert.Thumbprint
     $rsaFile = $WorkingCert.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
}
Catch
{
     "Error: unable to locate certificate for $($CertCN)"
     Exit
}

if ($WorkingCert.PrivateKey) {
     $WorkingCert.PrivateKey
}
else
{
     "No private key found"  
}

If you get No private key found message it means there is no private key in the MachineKeys folder. Even though certificate properties can claim otherwise (there is a key icon and message You have a private key that corresponds to this certificate). Although I don't know why but for some certificates above situation happens.

As a workaround, follow these steps:

1. Go to the local machine cert store and delete your certificate.
2. Import your certificate to the local user store first.
3. Then import your certificate to the local machine store.
4. Set access rights for Network Service user.

If you follow steps above, private key will be added to MachineKeys folder and error will disappear. Obviously you have to repeat these steps for every cluster node.