How to use ssl client certificate (p12) with Scrapy?

Question

I need to use client certificate file in format p12(PKCS12) to talk to a webserver with scrapy, is there a way to do that ?

Answer 1

I can't offer you a tested and complete solution here, but I know a few places where some adjustments might give you what you need.

The starting point is scrapy's ContextFactory object which defines the SSL/TLS configuration. The standard implementation ScrapyClientContextFactory doesn't use client certificates and also doesn't do any server certificate verification, it just accepts any certificate. (More details)

When looking into the source code however you see the alternative BrowserLikeContextFactory is creating an optionsForClientTLS object.

This object can also take a clientCertificate parameter to authenticate to the server. (Details)

So in theory you need to subclass BrowserLikeContextFactory, write there your own creatorForNetloc method and make it create optionsForClientTLS that also have a clientCertificate

In a gist:

@implementer(IPolicyForHTTPS)
class ClientCertContextFactory(BrowserLikeContextFactory):

    def creatorForNetloc(self, hostname, port):
        with open('yourcert.pem') as keyAndCert:
            myClientCert = twisted.internet.ssl.PrivateCertificate.load(keyAndCert.read())
        return optionsForClientTLS(hostname.decode("ascii"),
                                   trustRoot=platformTrust(),
                                   clientCertificate=myClientCert,
                                   extraCertificateOptions={
                                        'method': self._ssl_method,
                                   })

Activating the context factory in settings.py:

DOWNLOADER_CLIENTCONTEXTFACTORY = 'your.package.ClientCertContextFactory'

According to the docs twisted.internet.ssl.PrivateCertificate can only load pem or asn.1 format keys, means you will have to convert your key into pem format:

openssl pkcs12 -in client_ssl.pfx -out client_ssl.pem -clcerts

(Borrowed from Converting pfx to pem using openssl)

Update Conversion for PKCS12 files in p12 format:

openssl pkcs12 -in client_cert.p12 -out client_cert.pem -clcerts