curl with client certificate authentication

Question

We like to access a webserver using client certificate authentication instead of basic authentication.

Certificate is a PEM cert and the key file is a separate file.

The curl call looks like this:

curl -v --cert cert.crt --key key.key --pass foobar https://testserver/soap/request

From the debug log:

HTTP/1.1 401 Unauthorized
X-message-code: PWD_WRONG
WWW-Authenticate: Basic realm="Test Platform"
Transfer-Encoding: chunked
Date: Wed, 30 May 2018 10:26:51 GMT
Server: TEST
Set-Cookie: ...

I learned that WWW-Authenticate: Basic is a request from the server to the client to provide a Basic auth. However this may happen for an unauthenticated request, so it does not prove the server does not provide certificate authentication.

Reading https://medium.com/@sevcsik/authentication-using-https-client-certificates-3c9d270e8326 ...as double-check I tried with a Firefox Browser by merging both files to a PKCS#12 format and importing that into the browser. It works, so the server supports certificate authentication.

What is wrong about the failing curl call? Is there any test server supporting certificate auth? https://www.httpbin.org/ does not seem to provide that setup.

The same error occurs when using the PKCS#12 version with curl like this:

curl -v --cert cert.p12 --cert-type p12 --pass foobar
https://testserver/soap/request...

---

Complete debug log:

* Uses proxy env variable no_proxy == '10.0.0.0/8,127.0.0.1,172.16.0.0/12,192.168.20.0/24,'
* Uses proxy env variable https_proxy == 'http://gateway01:8080'
*   Trying 10.190.224.23...
* TCP_NODELAY set
* Connected to gateway01 (10.190.224.23) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to testserver:443
> CONNECT testserver:443 HTTP/1.1
> Host: testserver:443
> User-Agent: curl/7.60.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; ST=Baden-Wurttemberg; L=...; O=...; OU=...; CN=...
*  start date: Aug 15 07:54:38 2016 GMT
*  expire date: Aug 15 07:54:38 2019 GMT
*  subjectAltName: host "testserver" matched cert's "*.testserver"
*  issuer: C=NL; L=Amsterdam; O=Verizon Enterprise Solutions; OU=Cybertrust; CN=Verizon Public SureServer CA G14-SHA2
*  SSL certificate verify ok.
> GET /soap/request HTTP/1.1
> Host: testserver
> User-Agent: curl/7.60.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< X-message-code: PWD_WRONG
< WWW-Authenticate: Basic realm="Test Platform"
< Transfer-Encoding: chunked
< Date: Wed, 06 Jun 2018 05:10:56 GMT
< Server: TEST
< Set-Cookie: ...; path=/; httponly; secure
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< 
* Connection #0 to host gateway01 left intact

As Firefox is able to authenticate, I analyzed both requests with Wireshark. I find the certificate data by filtering for "tcp contains " and the known CN of my certificate in both network dumps, but Firefox and curl are placing the certs on different packets. Having a SOCKS5 proxy on localhost:8081 I capture loopback interface and filter on that tcp port 8081 which you can see in the dump:

Firefox:

$ tshark -r firefox_request.pcapng
    1 0.000000000    127.0.0.1 → 127.0.0.1    TCP 74 51254 → 8081 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1945282596 TSecr=0 WS=128
    2 0.000013339    127.0.0.1 → 127.0.0.1    TCP 74 8081 → 51254 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1945282596 TSecr=1945282596 WS=128
    3 0.000025766    127.0.0.1 → 127.0.0.1    TCP 66 51254 → 8081 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=1945282597 TSecr=1945282596
    4 0.000052377    127.0.0.1 → 127.0.0.1    TCP 69 51254 → 8081 [PSH, ACK] Seq=1 Ack=1 Win=43776 Len=3 TSval=1945282597 TSecr=1945282596
    5 0.000058018    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51254 [ACK] Seq=1 Ack=4 Win=43776 Len=0 TSval=1945282597 TSecr=1945282597
    6 0.000110299    127.0.0.1 → 127.0.0.1    TCP 68 8081 → 51254 [PSH, ACK] Seq=1 Ack=4 Win=43776 Len=2 TSval=1945282597 TSecr=1945282597
    7 0.000127024    127.0.0.1 → 127.0.0.1    TCP 66 51254 → 8081 [ACK] Seq=4 Ack=3 Win=43776 Len=0 TSval=1945282597 TSecr=1945282597
    8 0.000145980    127.0.0.1 → 127.0.0.1    TCP 112 51254 → 8081 [PSH, ACK] Seq=4 Ack=3 Win=43776 Len=46 TSval=1945282597 TSecr=1945282597 [TCP segment of a reassembled PDU]
    9 0.049958725    127.0.0.1 → 127.0.0.1    TCP 76 8081 → 51254 [PSH, ACK] Seq=3 Ack=50 Win=43776 Len=10 TSval=1945282646 TSecr=1945282597
   10 0.050239394    127.0.0.1 → 127.0.0.1    TCP 583 51254 → 8081 [PSH, ACK] Seq=50 Ack=13 Win=43776 Len=517 TSval=1945282647 TSecr=1945282646 [TCP segment of a reassembled PDU]
   11 0.100158230    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51254 [ACK] Seq=13 Ack=567 Win=44800 Len=0 TSval=1945282697 TSecr=1945282647
   12 0.107621822    127.0.0.1 → 127.0.0.1    TLSv1.2 2786 Server Hello
   13 0.107672270    127.0.0.1 → 127.0.0.1    TCP 66 51254 → 8081 [ACK] Seq=567 Ack=2733 Win=174720 Len=0 TSval=1945282704 TSecr=1945282704
   14 0.107832838    127.0.0.1 → 127.0.0.1    TLSv1.2 1426 Certificate, Server Key Exchange
   15 0.108417953    127.0.0.1 → 127.0.0.1    TCP 1426 8081 → 51254 [PSH, ACK] Seq=4093 Ack=567 Win=44800 Len=1360 TSval=1945282705 TSecr=1945282704 [TCP segment of a reassembled PDU]
   16 0.108423407    127.0.0.1 → 127.0.0.1    TCP 66 51254 → 8081 [ACK] Seq=567 Ack=5453 Win=436608 Len=0 TSval=1945282705 TSecr=1945282704
   17 0.109263897    127.0.0.1 → 127.0.0.1    TLSv1.2 2674 Certificate Request, Server Hello Done
   18 0.118830308    127.0.0.1 → 127.0.0.1    TCP 3613 51254 → 8081 [PSH, ACK] Seq=567 Ack=8061 Win=464896 Len=3547 TSval=1945282715 TSecr=1945282706 [TCP segment of a reassembled PDU]
   19 0.118851470    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51254 [ACK] Seq=8061 Ack=4114 Win=175744 Len=0 TSval=1945282715 TSecr=1945282715
   20 0.173415105    127.0.0.1 → 127.0.0.1    TLSv1.2 117 Change Cipher Spec, Client Hello[Malformed Packet]
   21 0.173932334    127.0.0.1 → 127.0.0.1    TCP 798 51254 → 8081 [PSH, ACK] Seq=4114 Ack=8112 Win=464896 Len=732 TSval=1945282770 TSecr=1945282770 [TCP segment of a reassembled PDU]
   22 0.173943923    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51254 [ACK] Seq=8112 Ack=4846 Win=182912 Len=0 TSval=1945282770 TSecr=1945282770
   23 0.250413758    127.0.0.1 → 127.0.0.1    TLSv1.2 844 Application Data
   24 0.300115138    127.0.0.1 → 127.0.0.1    TCP 66 51254 → 8081 [ACK] Seq=4846 Ack=8890 Win=464896 Len=0 TSval=1945282897 TSecr=1945282847
   25 2.490166002    127.0.0.1 → 127.0.0.1    TCP 798 51254 → 8081 [PSH, ACK] Seq=4846 Ack=8890 Win=464896 Len=732 TSval=1945285087 TSecr=1945282847 [TCP segment of a reassembled PDU]
   26 2.490201268    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51254 [ACK] Seq=8890 Ack=5578 Win=184320 Len=0 TSval=1945285087 TSecr=1945285087
   27 2.554148656    127.0.0.1 → 127.0.0.1    TLSv1.2 844 Application Data
   28 2.554155780    127.0.0.1 → 127.0.0.1    TCP 66 51254 → 8081 [ACK] Seq=5578 Ack=9668 Win=464896 Len=0 TSval=1945285151 TSecr=1945285151
   29 12.610141935    127.0.0.1 → 127.0.0.1    TCP 66 [TCP Keep-Alive] 51254 → 8081 [ACK] Seq=5577 Ack=9668 Win=464896 Len=0 TSval=1945295207 TSecr=1945285151
   30 12.610162448    127.0.0.1 → 127.0.0.1    TCP 66 [TCP Keep-Alive ACK] 8081 → 51254 [ACK] Seq=9668 Ack=5578 Win=184320 Len=0 TSval=1945295207 TSecr=1945285151

Curl:

$ tshark -r curl_request.pcapng
    1 0.000000000    127.0.0.1 → 127.0.0.1    TCP 74 51196 → 8081 [SYN] Seq=0 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1944382941 TSecr=0 WS=128
    2 0.000014517    127.0.0.1 → 127.0.0.1    TCP 74 8081 → 51196 [SYN, ACK] Seq=0 Ack=1 Win=43690 Len=0 MSS=65495 SACK_PERM=1 TSval=1944382941 TSecr=1944382941 WS=128
    3 0.000026917    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=1 Ack=1 Win=43776 Len=0 TSval=1944382941 TSecr=1944382941
    4 0.000068294    127.0.0.1 → 127.0.0.1    TCP 69 51196 → 8081 [PSH, ACK] Seq=1 Ack=1 Win=43776 Len=3 TSval=1944382941 TSecr=1944382941
    5 0.000076382    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=1 Ack=4 Win=43776 Len=0 TSval=1944382941 TSecr=1944382941
    6 0.000105767    127.0.0.1 → 127.0.0.1    TCP 68 8081 → 51196 [PSH, ACK] Seq=1 Ack=4 Win=43776 Len=2 TSval=1944382941 TSecr=1944382941
    7 0.000114276    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=4 Ack=3 Win=43776 Len=0 TSval=1944382941 TSecr=1944382941
    8 0.012753479    127.0.0.1 → 127.0.0.1    TCP 76 51196 → 8081 [PSH, ACK] Seq=4 Ack=3 Win=43776 Len=10 TSval=1944382954 TSecr=1944382941
    9 0.056073156    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=3 Ack=14 Win=43776 Len=0 TSval=1944382997 TSecr=1944382954
   10 0.060558849    127.0.0.1 → 127.0.0.1    TCP 76 8081 → 51196 [PSH, ACK] Seq=3 Ack=14 Win=43776 Len=10 TSval=1944383002 TSecr=1944382954
   11 0.106077184    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=14 Ack=13 Win=43776 Len=0 TSval=1944383047 TSecr=1944383002
   12 0.156827822    127.0.0.1 → 127.0.0.1    TLSv1 583 Client Hello
   13 0.156838865    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=13 Ack=531 Win=44800 Len=0 TSval=1944383098 TSecr=1944383098
   14 0.204829368    127.0.0.1 → 127.0.0.1    TLSv1.2 2786 Server Hello
   15 0.204837221    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=531 Ack=2733 Win=174720 Len=0 TSval=1944383146 TSecr=1944383146
   16 0.204942723    127.0.0.1 → 127.0.0.1    TLSv1.2 4146 Certificate, Server Key Exchange
   17 0.204950957    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=531 Ack=6813 Win=305664 Len=0 TSval=1944383146 TSecr=1944383146
   18 0.204981363    127.0.0.1 → 127.0.0.1    TLSv1.2 1314 Certificate Request, Server Hello Done
   19 0.204987399    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=531 Ack=8061 Win=436608 Len=0 TSval=1944383146 TSecr=1944383146
   20 0.208331980    127.0.0.1 → 127.0.0.1    TLSv1.2 2066 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
   21 0.208341083    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=8061 Ack=2531 Win=175744 Len=0 TSval=1944383149 TSecr=1944383149
   22 0.255529146    127.0.0.1 → 127.0.0.1    TLSv1.2 117 Change Cipher Spec, Client Hello[Malformed Packet]
   23 0.255792019    127.0.0.1 → 127.0.0.1    TLSv1.2 220 Application Data
   24 0.255799639    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=8112 Ack=2685 Win=179840 Len=0 TSval=1944383197 TSecr=1944383197
   25 2.312629664    127.0.0.1 → 127.0.0.1    TLSv1.2 535 Application Data
   26 2.312809015    127.0.0.1 → 127.0.0.1    TLSv1.2 97 Encrypted Alert
   27 2.312821190    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=8581 Ack=2716 Win=179840 Len=0 TSval=1944385254 TSecr=1944385254
   28 2.313444679    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [FIN, ACK] Seq=2716 Ack=8581 Win=444800 Len=0 TSval=1944385255 TSecr=1944385254
   29 2.356079624    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [ACK] Seq=8581 Ack=2717 Win=179840 Len=0 TSval=1944385297 TSecr=1944385255
   30 2.357676736    127.0.0.1 → 127.0.0.1    TCP 66 8081 → 51196 [FIN, ACK] Seq=8581 Ack=2717 Win=179840 Len=0 TSval=1944385299 TSecr=1944385255
   31 2.357690888    127.0.0.1 → 127.0.0.1    TCP 66 51196 → 8081 [ACK] Seq=2717 Ack=8582 Win=444800 Len=0 TSval=1944385299 TSecr=1944385299

Filtering for the special CN of my local certificate I can find this in both dumps:

$ for file in {firefox,curl}_request.pcapng;do tshark -r "$file" -Y "tcp contains FOOBAR";done
   18 0.118830308    127.0.0.1 → 127.0.0.1    TCP 3613 51254 → 8081 [PSH, ACK] Seq=567 Ack=8061 Win=464896 Len=3547 TSval=1945282715 TSecr=1945282706 [TCP segment of a reassembled PDU]
   20 0.208331980    127.0.0.1 → 127.0.0.1    TLSv1.2 2066 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message

However it is detected as part of the TLS handshake only for curl:

$ for file in {firefox,curl}_request.pcapng;do tshark -r "$file" -Y "ssl contains FOOBAR";done
   20 0.208331980    127.0.0.1 → 127.0.0.1    TLSv1.2 2066 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message

What is different in the authentication of Firefox (successful) and Curl (failing)?

Answer 1

The server includes a list of acceptable certificate authorities in its CertificateRequest message. The client should then send a certificate chain that is acceptable according to those criteria.

Based on the fact that your client certificate is included in a "TCP segment of a reassembled PDU" in Firefox, I guess that it additionally included intermediate certificate authorities to complete the certification path. This enlarges the handshake message such that it no longer fits in a single TLS record (or TCP segment).

With cURL, you have probably not included this intermediate certificate which results in a smaller Certificate message from the client to server that fits in a single TLS record. This then results in Wireshark displaying something for ssl contains YOUR_CN. (The reason for this display issue is actually a missing feature in Wireshark, reassembly of handshake messages across different TLS records, bug 3303.)

To fix this, you can try appending the intermediate CAs to your local client certificate.

(I could be mistaken, but this is one possible reason I can think of based on the text trace alone.)

Answer 2

Thanks Lekensteyn, you were right, it was about CA intermediate certs. Looking inside the Certificate, Client Key Exchange of Firefox, it was adding another intermediate CA from Comodo, I just don't know where it got that from.

However merging the 2 certs into one file didn't help in curl, neither helped adding 2 --cert. But --cacert worked which is interesting as --cacert is actually meant to set the CA cert to check the server certs on client side.