OpenSSL.crypto.X509.sign() throws " 'bytes' object has no attribute 'encode' "

Question

So I'm trying to use the OpenSSL crypto module to generate a new CA certificate with this code:

#warning: this block is background information, probably not 
#where my real problem is

#generate the key pair
key=OpenSSL.crypto.PKey()
key.generate_key(OpenSSL.crypto.TYPE_RSA,2048)

#print the private and public keys as PEMs
print(codecs.decode(OpenSSL.crypto.dump_publickey(OpenSSL.crypto.FILETYPE_PEM,key),'utf8'))
print(codecs.decode(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM,key),'utf8'))

#generate a new x509 certificate
ca=OpenSSL.crypto.X509()

#fill it with goodies
ca.set_version(3)
ca.set_serial_number(1)
ca.get_subject().CN = "CA.test.com"
ca.gmtime_adj_notBefore(0)
ca.gmtime_adj_notAfter(60 * 60 * 24 * 365 * 10)
ca.set_issuer(ca.get_subject())
ca.set_pubkey(key)

#print the new certificate as a PEM
print(codecs.decode(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM,ca),'utf8'))

The certificate that prints out decodes OK at the SSLShopper certificate decoder so I'm feeling pretty confident about that part. The trouble really starts when I try to sign the certificate with

ca.sign(key, 'sha1')

because I get an " expected type 'bytes', got 'str' instead " from the IDE. Check the OpenSSL.crypto.X509.sign() documentation and confirm it really expects a bytes object, switch to

digestname='sha1'.encode('utf-8')
ca.sign(key, digestname)

and I get an " AttributeError: 'bytes' object has no attribute 'encode' " exception. Stepping through the code I find the exception is thrown in OpenSSL._util.byte_string() because

if PY3:
    def byte_string(s):
    return s.encode("charmap")
else:
    def byte_string(s):
    return s

where PY3=True and s={bytes}b'sha1', which of course has no .encode method.

Thus began my demoralizing 'bytes' vs 'str' struggle. I'd like to think I'm not the only one to be having this problem but my very best Google-fu has convinced me otherwise. At this point I don't even know what to go read about to get this one figured out.

Answer 1

It turns out that my IDE (PyCharm) was leading me astray. ca.sign(key, 'sha1') is really the correct way to do it. Even though PyCharm gives a type error program execution flows right through the statement and the output is correct.