OpenSSL 1.0.1 handshake workaround in Ubuntu? [closed]

Question

I've encountered a serious bug in OpenSSL 1.0.1 on Ubuntu 12.04:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665452

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666051 <- dated Oct 3 2012!

The gist of it is that I'm able to connect to some servers but not others. Connecting to google works:

openssl s_client -connect mail.google.com:443 -debug -state -msg -CAfile /etc/ssl/certs/ca-certificates.crt

...
Protocol  : TLSv1.1
Cipher    : ECDHE-RSA-RC4-SHA
Session-ID: 94DB1AC8531115C501434B16A5E9B735722768581778E4FEA4D9B19988551397
Session-ID-ctx:
Master-Key: 8694BF510CD7568CBAB39ECFD32D115C511529871F3030B67A4F7AEAF957D714D3E94E4CE6117F686C975EFF21FB8708
Key-Arg   : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 100800 (seconds)
TLS session ticket:
0000 - fb 52 d6 d3 3c a8 75 e1-1f 1d f6 23 ab ce 55 44   .R..<.u....#..UD
0010 - 27 bf ad c4 7a 0d 83 c8-48 59 48 4b 39 bb 3c c7   '...z...HYHK9.<.
0020 - 01 1e ad b3 13 de 65 d4-e8 ea e4 35 89 83 55 8e   ......e....5..U.
0030 - e4 d5 9f 60 58 51 33 9b-83 34 b9 35 3d 46 cb a3   ...`XQ3..4.5=F..
0040 - 35 7b 48 5d 7b 86 5c d5-a1 14 9d 8c 3e 93 eb fb   5{H]{.\.....>...
0050 - ac 78 75 72 9b d2 bc 67-f2 fa 5b 75 80 a6 31 d8   .xur...g..[u..1.
0060 - 71 15 85 7f 55 4d dc fb-b0 b5 33 db 6d 36 8c c6   q...UM....3.m6..
0070 - e8 f9 54 7a 29 69 87 2c-dd f3 c5 cf 26 55 6f 6e   ..Tz)i.,....&Uon
0080 - 45 73 7a 1d e4 b3 be b2-92 3f 0b ed c4 1c a5 24   Esz......?.....$
0090 - 3c f0 ca a5                                       <...

Start Time: 1354063165
Timeout   : 300 (sec)
Verify return code: 0 (ok)

But connecting to facebook doesn't:

openssl s_client -connect graph.facebook.com:443 -debug -state -msg -CAfile /etc/ssl/certs/ca-certificates.crt -cipher SRP-AES-256-CBC-SHA

CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0xddd2c0 [0xddd340] (64 bytes => 64 (0x40))
0000 - 16 03 01 00 3b 01 00 00-37 03 02 50 b5 5d 75 42   ....;...7..P.]uB
0010 - c2 78 55 49 b5 2e de 4f-00 a6 a8 d5 cf 10 92 44   .xUI...O.......D
0020 - 28 62 34 d6 61 5e 88 c3-68 8b 96 00 00 04 c0 20   (b4.a^..h......
0030 - 00 ff 02 01 00 00 09 00-23 00 00 00 0f 00 01 01   ........#.......
>>> TLS 1.1  [length 003b]
    01 00 00 37 03 02 50 b5 5d 75 42 c2 78 55 49 b5
    2e de 4f 00 a6 a8 d5 cf 10 92 44 28 62 34 d6 61
    5e 88 c3 68 8b 96 00 00 04 c0 20 00 ff 02 01 00
    00 09 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state
read from 0xddd2c0 [0xde28a0] (7 bytes => 7 (0x7))
0000 - 15 03 02 00 02 02 28                              ......(
SSL3 alert read:fatal:handshake failure
<<< TLS 1.1  [length 0002]
    02 28
SSL_connect:error in unknown state
140581179446944:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:724:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 64 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE

The facebook connection either hangs after the client sends its hello buffer and never receives the server hello response, or returns with an error code if I pass in a cipher it recognizes. This happens with both -tls1 and -ssl3. I've tried every parameter to openssl I can think of.

apt-cache showpkg openssl

...
Provides:
1.0.1-4ubuntu5.5 -
1.0.1-4ubuntu5.3 -
1.0.1-4ubuntu3 -

I've also tried every parameter I can think of to curl but with no success, because it uses openssl under the hood.

I'm concerned that Ubuntu can't establish secure connections (an astounding statement, I realize). After two solid days of beating my head against this problem, I'm basically praying at this point that someone knows a workaround. I'm considering a downgrade to OpenSSL 1.0.0 or using libcurl4-dev with gnutls-dev instead. Both solutions leave a rotten taste in my mouth. Thanks in advance for any help you can provide.

P.S. all of this work is so that my server can interface with external https REST APIs. I consider this a fundamental requirement in any webserver today, no excuses.

UPDATE: Here is my output without passing a cipher. It doesn't matter if I pass -CAfile or not either:

openssl s_client -connect graph.facebook.com:443 -debug -state -msg -CAfile /etc/ssl/certs/ca-certificates.crt

CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x14ed1a0 [0x1515bf0] (226 bytes => 226 (0xE2))
0000 - 16 03 01 00 dd 01 00 00-d9 03 02 50 b6 39 78 6a   ...........P.9xj
0010 - 24 95 8e dc 62 19 37 4b-ab 77 b8 66 cd 48 ba a2   $...b.7K.w.f.H..
0020 - a1 2a f8 1d f8 c9 5d fb-9d db 84 00 00 66 c0 14   .*....]......f..
0030 - c0 0a c0 22 c0 21 00 39-00 38 00 88 00 87 c0 0f   ...".!.9.8......
0040 - c0 05 00 35 00 84 c0 12-c0 08 c0 1c c0 1b 00 16   ...5............
0050 - 00 13 c0 0d c0 03 00 0a-c0 13 c0 09 c0 1f c0 1e   ................
0060 - 00 33 00 32 00 9a 00 99-00 45 00 44 c0 0e c0 04   .3.2.....E.D....
0070 - 00 2f 00 96 00 41 c0 11-c0 07 c0 0c c0 02 00 05   ./...A..........
0080 - 00 04 00 15 00 12 00 09-00 14 00 11 00 08 00 06   ................
0090 - 00 03 00 ff 02 01 00 00-49 00 0b 00 04 03 00 01   ........I.......
00a0 - 02 00 0a 00 34 00 32 00-0e 00 0d 00 19 00 0b 00   ....4.2.........
00b0 - 0c 00 18 00 09 00 0a 00-16 00 17 00 08 00 06 00   ................
00c0 - 07 00 14 00 15 00 04 00-05 00 12 00 13 00 01 00   ................
00d0 - 02 00 03 00 0f 00 10 00-11 00 23 00 00 00 0f 00   ..........#.....
00e0 - 01 01                                             ..
>>> TLS 1.1  [length 00dd]
    01 00 00 d9 03 02 50 b6 39 78 6a 24 95 8e dc 62
    19 37 4b ab 77 b8 66 cd 48 ba a2 a1 2a f8 1d f8
    c9 5d fb 9d db 84 00 00 66 c0 14 c0 0a c0 22 c0
    21 00 39 00 38 00 88 00 87 c0 0f c0 05 00 35 00
    84 c0 12 c0 08 c0 1c c0 1b 00 16 00 13 c0 0d c0
    03 00 0a c0 13 c0 09 c0 1f c0 1e 00 33 00 32 00
    9a 00 99 00 45 00 44 c0 0e c0 04 00 2f 00 96 00
    41 c0 11 c0 07 c0 0c c0 02 00 05 00 04 00 15 00
    12 00 09 00 14 00 11 00 08 00 06 00 03 00 ff 02
    01 00 00 49 00 0b 00 04 03 00 01 02 00 0a 00 34
    00 32 00 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09
    00 0a 00 16 00 17 00 08 00 06 00 07 00 14 00 15
    00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0f
    00 10 00 11 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state

Answer 1

Why are you passing -cipher SRP-AES-256-CBC-SHA when connecting to graph.facebook.com? Facebook certainly doesn't support SRP: http://srp.stanford.edu/.

Does it work if you don't pass that?

Also, can you give the IP address that you're getting? With 69.171.229.17, I can reproduce that exact ClientHello (modulo the nonce and with RC4-SHA are the only cipher save the SCSV) and I get a successful handshake.

Lastly, have you tried doing over an SSH tunnel to somewhere else? Sadly, when deploying TLS features in Chrome we have repeatedly found networking hardware that breaks TLS connections. (Although I can't think of a case where -ssl3 wouldn't fix it unless the hardware was actively trying to censor connections.)

Answer 2

Setting the MTU on my Ubuntu box from 1500 to 1496 (due to one of our firewalls being set too low) allows me to receive a response from the server without having to reboot (be sure to call ifconfig first and write down your original MTU which should be 1500):

sudo ifconfig eth0 mtu 1496

I discovered my MTU by pinging with successively larger buffers (add 28 bytes for UDP header):

Fails for 1472 + 28 = 1500:

ping -s 1472 facebook.com
PING facebook.com (66.220.158.16) 1472(1500) bytes of data.
...

Works for 1468 + 28 = 1496:

ping -s 1468 facebook.com
PING facebook.com (69.171.229.16) 1468(1496) bytes of data.
1476 bytes from www-slb-ecmp-06-prn1.facebook.com (69.171.229.16): icmp_req=1 ttl=242 time=30.0 ms
...

With 1496 I'm now able to curl to facebook.com:

curl -v https://facebook.com
* About to connect() to facebook.com port 443 (#0)
*   Trying 66.220.152.16... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=www.facebook.com
*        start date: 2012-06-21 00:00:00 GMT
*        expire date: 2013-12-31 23:59:59 GMT
*        subjectAltName: facebook.com matched
*        issuer: O=VeriSign Trust Network; OU=VeriSign, Inc.; OU=VeriSign International Server CA - Class 3; OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: facebook.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https://www.facebook.com/
< Content-Type: text/html; charset=utf-8
< X-FB-Debug: 3vAg1O5OG9hB/EWC+gk1Kl3WLJRGmlQDaEodirWb+i0=
< Date: Wed, 28 Nov 2012 19:52:25 GMT
< Connection: keep-alive
< Content-Length: 0
<
* Connection #0 to host facebook.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

I personally think that MTU should have absolutely nothing to do with what the user sees at the stream level with TCP so I hope the OpenSSL folks fix this. I also wish that someone would invent an automagic bug submitter for bugs that are profoundly widespread and time-sucking.