Using Security Transforms to verify an RSA signature created with Ruby/OpenSSL

Question

I'm trying to implement a simple license key scheme for my app and I'm running into significant roadblocks. I'm following the example at OpenSSL for License Keys.

Since that blog post was written in 2004 and OpenSSL has been deprecated on OS X I'm attempting to use the Security Transforms API to accomplish the license key verification instead of OpenSSL. I'm generating the private and public keys with OpenSSL, however; the license key is generated using the private key by a Ruby web app using the Ruby OpenSSL wrapper library from a SHA-256 digest of the purchaser's email address.

The problem is that nothing I do seems to produce a signature from Ruby using OpenSSL that the Security Transforms API will verify.

The Ruby code I'm working off of is:

require('openssl')

# The email address used as the content of the license key.
license = '[email protected]'

# Generate the public/private keypair.
`openssl genrsa -out private_key.pem 2048`
`openssl rsa -in conductor.pem -out public_key.data -pubout`

# Get the private key and a hash of the license.
private_key = OpenSSL::PKey::RSA.new(File.read('private_key.pem'))
signature   = OpenSSL::Digest::SHA256.digest(license)

# The signature passed to SecVerifyTransformCreate in the OS X app. I'm not sure which of these SecVerifyTransformCreate is expecting (the binary digest, a hex representation of the digest, or the original un-digested content), but none of them work.
signature_out = signature
#signature_out = OpenSSL::Digest::SHA256.hexdigest(license)
#signature_out = license

File.write('signature.data', signature_out)

# Sign the email address to generate the license key. Using the OpenSSL::PKey::PKey#sign method produces a license key that can only be verified on the command line by running:
#
#   echo -n [email protected] | openssl dgst -sha256 -sign test.pem
#
# while using the #private_encrypt method produces a key that can only be verified on the command line by running:
#
#   echo -n [email protected] | openssl dgst -sha256 -binary | openssl rsautl -sign -inkey test.pem
#
# I'm not sure what the exact difference between the two commands above is and why they correspond to the two different Ruby signing methods below. Neither approach produces something that SecVerifyTransformCreate will verify, however.
File.write('license_key.data',
           private_key.sign(OpenSSL::Digest::SHA256.new, license))
#           private_key.private_encrypt(signature))

And the corresponding verification code in Objective-C:

// Get the data.
NSData *publicKeyData  = [NSData dataWithContentsOfFile:@"public_key.data"];
NSData *signatureData  = [NSData dataWithContentsOfFile:@"signature.data"];
NSData *licenseKeyData = [NSData dataWithContentsOfFile:@"license_key.data"];

// Import the public key.
SecItemImportExportKeyParameters keyParameters = {};
SecExternalFormat format = kSecFormatOpenSSL;
SecExternalItemType type = kSecItemTypePublicKey;
CFArrayRef publicKeys;

SecItemImport((__bridge CFDataRef)publicKeyData,
              NULL,
              &format,
              &type,
              0,
              &keyParameters,
              NULL,
              &publicKeys);

NSArray *publicKeysArray = (__bridge_transfer NSArray *)publicKeys;
SecKeyRef publicKey = (__bridge SecKeyRef)publicKeysArray[0]; // TODO: How do we need to bridge this return value?

CFErrorRef error = NULL;

SecTransformRef verifier = SecVerifyTransformCreate(publicKey, (__bridge CFDataRef)signatureData, &error);

SecTransformSetAttribute(verifier, kSecTransformDebugAttributeName, kCFBooleanTrue, &error);
SecTransformSetAttribute(verifier, kSecTransformInputAttributeName, (__bridge CFDataRef)licenseKeyData, &error);
SecTransformSetAttribute(verifier, kSecDigestTypeAttribute, kSecDigestSHA2, &error);
SecTransformSetAttribute(verifier, kSecDigestLengthAttribute, (__bridge CFNumberRef)@256, &error);

// I'm not sure if one of these transform attributes is necessary, but neither of them produces a verified result anyways.
//  SecTransformSetAttribute(verifier, kSecInputIsAttributeName, kSecInputIsDigest, &error);
//  SecTransformSetAttribute(verifier, kSecInputIsAttributeName, kSecInputIsRaw, &error);

NSNumber *result = (__bridge NSNumber *)SecTransformExecute(verifier, &error);

NSLog(@"Result: %@", result);

Does anyone know how I can make this work? I've literally spent days getting to the point where I'm at now and I've exhausted my ability to debug this any further, so if anyone has any insight it would be hugely appreciated!

Answer 1

In short you're mixing up some key concepts. Here's a quick primer on how this works.

1. A document (your license data/email) is hashed with a digest (SHA256)
2. Private key encrypts the hash. This is the binary signature.
3. The binary signature needs to be encoded into a format convenient for transport, usually to text with base64 or something similar.
4. The encoded signature is passed over with document to verifying party (your objc application) for verification
5. Document (your license) is hashed again with same digest (SHA256)
6. Encoded signature is decoded back into binary
7. Public key decrypts signature which reveals the original hash
8. This decrypted hash is compared with calculated one, if they match document is verified.

On the ruby side you're confusing signature and document. You need to hash the license with SHA256 and then encrypt that with private key to produce a signature. You're just saving a hash of the document as the signature. Try this out on ruby side of things:

require 'openssl'
require 'base64'

license = '[email protected]'
private_key = OpenSSL::PKey::RSA.new(File.read('private_key.pem'))
digest  = OpenSSL::Digest::SHA256.new
signature = private_key.sign digest, license
signature_out = Base64.encode64(signature)

File.write('signature.data', signature_out)
File.write('license_key.data', license) # no hash, no signing

The ruby docs around this can be found here.

I'm not super familiar with the library you're using on Objective-C side of things, but the trick here is to make sure you're using the same Digest algorithm for hashing on both ends (SHA256), check same Encryption algorithm (RSA) and the public key and private key are compatible (matching RSA modulus and public exponents), and same encoding for the binary signature data passed back and forth (base64, hex, etc)

On the ruby side you're generating a signature with SHA256 and on the objective-c it looks like you're verifying it with SHA-2 size 256 so that looks ok.

Decode the signature (if you're writing binary from ruby you can skip this)

SecTransformRef decoder = SecDecodeTransformCreate(kSecBase64Encoding, &error);
if (error) { CFShow(error); exit(-1); }

SecTransformSetAttribute(decoder, 
                         kSecTransformInputAttributeName,
                         signatureData, 
                         &error);
if (error) { CFShow(error); exit(-1); }

CFDataRef signature = SecTransformExecute(decoder, &error);
if (error) { CFShow(error); exit(-1); }

For the verification you want something like this, gleamed from here:

verifier = SecVerifyTransformCreate(publicKey, signature, &error);
if (error) { CFShow(error); exit(-1); } // show your errors!

SecTransformSetAttribute(verifier,
                         kSecTransformInputAttributeName,
                         cfLicense,  // Converted from NSData
                         &error);
if (error) { CFShow(error); exit(-1); }

SecTransformSetAttribute(verifier, 
                         kSecDigestTypeAttribute, 
                         kSecDigestSHA2, 
                         &error);
if (error) { CFShow(error); exit(-1); }

SecTransformSetAttribute(verifier, 
                         kSecDigestLengthAttribute, 
                         (__bridge CFNumberRef)@256, 
                         &error);
if (error) { CFShow(error); exit(-1); }

result = SecTransformExecute(verifier, &error);
if (error) { CFShow(error); exit(-1); }

if (result == kCFBooleanTrue) {
  /* Signature was valid. */
} else {
  /* Signature was invalid. */
}