Cryptography: Why am I getting different RSA signatures depending on which certificate store the certificate was loaded from?

Question

I have some working code which produces a correct signature of a string if I load a certificate from a file or from the current user's store. However, if I load the exact same certificate (same .p12 and same thumbprint) from the Machine certificate store, it behaves differently. When loaded from that store, the signatures generated by my C# code are half the length (1024 bits instead of 2048) and are incorrect. The private key appears to be loading properly in both cases.

Why does which store the certificate is loaded from make any difference to which signature is generated? And why would the signature be half the length?

Loaded from CurrentUser:

Thumbprint: FBBE05A1C5F2AEF637CDE20A7985CD1011861651
Has private key:True
rsa.KeySize (bits) =2048
Signature Length (bits): 2048
Signature: kBC2yh0WCo/AU8aVo+VUbRoh67aIJ7SWM4dRMkNvt...

(correct)

Loaded from LocalMachine:

Thumbprint: FBBE05A1C5F2AEF637CDE20A7985CD1011861651
Has private key: True
rsa.KeySize (bits) = 1024
Signature Length (bits): 1024
Signature: RijmdQ73DXHK1IUYkOzov2R+WRdHW8tLqsH....

(incorrect - and note the 1024 bit key size and signature length)

Here's the C# I'm using:

        string s = "AE0DE01564,1484821101811,http://localhost:8080/example_site/CallBack";

        var inputData = Encoding.UTF8.GetBytes(s);

        var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
        store.Open(OpenFlags.ReadOnly | OpenFlags.OpenExistingOnly);

        string thumbprint = CleanThumbPrint("fb be 05 a1 c5 f2 ae f6 37 cd e2 0a 79 85 cd 10 11 86 16 51");
        X509Certificate2Collection col = store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, false);

        // TODO: close store.
        X509Certificate2 certificate = null;

        Console.WriteLine("Cert count: " + col.Count);
        if (col.Count == 1)
        {
            certificate = col[0];
            RSACryptoServiceProvider rsa = (RSACryptoServiceProvider)col[0].PrivateKey;

            // Force use of the Enhanced RSA and AES Cryptographic Provider with openssl-generated SHA256 keys
            var enhCsp = new RSACryptoServiceProvider().CspKeyContainerInfo;

            var cspparams = new CspParameters(enhCsp.ProviderType, enhCsp.ProviderName, rsa.CspKeyContainerInfo.KeyContainerName);

            rsa = new RSACryptoServiceProvider( cspparams);
            Console.WriteLine("Name: " + certificate.SubjectName.Name);
            Console.WriteLine("Thumbprint: " + certificate.Thumbprint);
            Console.WriteLine("Has private key: " + certificate.HasPrivateKey);
            Console.WriteLine("Sig algorithm: " + certificate.SignatureAlgorithm);
            Console.WriteLine("rsa.KeySize (bits) =" + rsa.KeySize);

            var sha256 = CryptoConfig.CreateFromName("SHA256");
            byte[] signature = rsa.SignData(inputData, sha256);

            Console.WriteLine("Signature Length (bits): " + signature.Length * 8);
            Console.WriteLine("Signature: " + System.Convert.ToBase64String(signature));
            Console.WriteLine();
      }

Answer 1

It turns out it was something to do with the file format of the certificate I was using which I created with OpenSSL and the fact that the crypto provider wasn't set. The critical command is number 5 below:

Here are the commands I used to create the working certificate:

1. Generate a keypair:

openssl genrsa -out private_key.pem 2048

2. Extract the public key:

openssl rsa -pubout -in private_key.pem -out public_key.pem

3. Create a CSR Certificate Signing Request from private key:

openssl req -new -key private_key.pem -out csr.csr

4. Generate a self signed certificate:

openssl x509 -req -days 1095 -in csr.csr -signkey private_key.pem -out certificate.crt

5. Create a PFX format certificate with the specified CSP:

openssl pkcs12 -export -in certificate.crt -inkey private_key.pem -CSP "Microsoft Enhanced RSA and AES Cryptographic Provider" -out TEST_pfx.pfx