Using MySqlConnection (.NET) to SSL connect to MySql-server fails from certain clients

Question

This is really a question about SSL, and Socket-traces are below, so don't be shy to continue reading even if you've never used MySql before. Anyway we're trying to connect to a MySql-server over SSL (with a Client .PFX certificate) using MySqlConnection (both .NET 2.0 & 4.0, version doesn't matter), as so:

using (MySqlConnection con = new MySqlConnection(@"server=xxx.yyy.zzz.uuu;database=whopper;user=Username;password=Secret;Certificate Store Location=CurrentUser;Pooling=false;SSL Mode=Required;Certificate Thumbprint=NNNN;Keepalive=60")) {
    con.Open();
}

The certificate is installed in the Personal store, and this whole thing works on some machines; but not on others. StackTrace:

System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at MySql.Data.MySqlClient.NativeDriver.StartSSL() in D:\Downloads\mysql-connector-net-6.4.4-src\Source\MySql.Data\NativeDriver.cs:line 353
   at MySql.Data.MySqlClient.NativeDriver.Open() in D:\Downloads\mysql-connector-net-6.4.4-src\Source\MySql.Data\NativeDriver.cs:line 267
   at MySql.Data.MySqlClient.Driver.Open() in D:\Downloads\mysql-connector-net-6.4.4-src\Source\MySql.Data\Driver.cs:line 211
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings) in D:\Downloads\mysql-connector-net-6.4.4-src\Source\MySql.Data\Driver.cs:line 199
   at MySql.Data.MySqlClient.MySqlConnection.Open() in D:\Downloads\mysql-connector-net-6.4.4-src\Source\MySql.Data\Connection.cs:line 489
   at ConsoleApplication1.Program.Main(String[] args) in D:\Downloads\mysql-connector-net-6.4.4-src\Source\ConsoleApplication1\Program.cs:line 14

Now, to debug this I've downloaded the latest src of MySqlConnection, so I can follow everything until the call to AuthenticateAsClient - where MySqlConnection calls the .NET framework to setup the SSL connection. Next I added the following to my app.config (to be get a socket trace and continue the investigation):

<?xml version="1.0"?>
<configuration>
    <system.diagnostics>
        <sources>
            <source name="System.Net.Sockets">
                <listeners>
                    <add name="Sockets"/>
                </listeners>
            </source>
        </sources>
        <switches>
            <add name="System.Net.Sockets" value="31"/>
        </switches>
        <sharedListeners>
            <add name="Sockets" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\socketdump.log"/>
        </sharedListeners>
        <trace autoflush="true"/>
    </system.diagnostics>
</configuration>

On the client machine where the connection accually succeeds, that file begins with:

System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Socket(InterNetwork#2)
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Socket() 
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::BeginConnect(142:3306#-1905378776)
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::BeginConnectEx()
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::InternalBind(0:0#0)
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::InternalBind() 
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::BeginConnectEx()    -> ConnectOverlappedAsyncResult#9029417
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::EndConnect(ConnectOverlappedAsyncResult#9029417)
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::EndConnect() 
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 3D 00 00 00 0A 35 2E 31-2E 35 34 2D 31 75 62 75 : =....5.1.54-1ubu
System.Net.Sockets Verbose: 0 : [5000] 00000010 : 6E 74 75 34 00 5E 04 00-00 2F 32 4C 69 4D 26 73 : ntu4.^.../2LiM&s
System.Net.Sockets Verbose: 0 : [5000] 00000020 : 32 00 FF FF 08 02 00 00-00 00 00 00 00 00 00 00 : 2...............
System.Net.Sockets Verbose: 0 : [5000] 00000030 : 00 00 00 00 3E 23 31 6B-4B 75 2B 72 7D 3F 60 48 : ....>#1kKu+r}?`H
System.Net.Sockets Verbose: 0 : [5000] 00000040 : 00                                              : .
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 65#65
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Send()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Send
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 04 00 00 01 8F AA 03 00-                        : ........
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Send()  -> 8#8
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Send()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Send
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 16 03 01 00 5A 01 00 00-56 03 01 4E 89 D2 2C 88 : ....Z...V..N..,.
System.Net.Sockets Verbose: 0 : [5000] 00000010 : 6E E9 57 C7 A0 A2 83 59-DC B1 1D 11 01 F6 32 F0 : n.W....Y......2.
System.Net.Sockets Verbose: 0 : [5000] 00000020 : E0 50 63 37 BD EE AD A2-07 3B CC 00 00 18 00 2F : .Pc7.....;...../
System.Net.Sockets Verbose: 0 : [5000] 00000030 : 00 35 00 05 00 0A C0 13-C0 14 C0 09 C0 0A 00 32 : .5.............2
System.Net.Sockets Verbose: 0 : [5000] 00000040 : 00 38 00 13 00 04 01 00-00 15 FF 01 00 01 00 00 : .8..............
System.Net.Sockets Verbose: 0 : [5000] 00000050 : 0A 00 06 00 04 00 17 00-18 00 0B 00 02 01 00    : ...............
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Send()  -> 95#95
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 16 03 01 00 4A                                  : ....J
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000005 : 02 00 00 46 03 01 D3 49-C4 02 A3 F4 60 21 C1 F0 : ...F...I....`!..
System.Net.Sockets Verbose: 0 : [5000] 00000015 : 19 06 33 F1 F2 9B D1 8D-65 C6 A6 EF C8 C8 21 BC : ..3.....e.....!.
System.Net.Sockets Verbose: 0 : [5000] 00000025 : C4 80 AC 9D 0D 0C 20 D9-5D A0 92 8A F5 70 D8 B0 : ...... .]....p..
System.Net.Sockets Verbose: 0 : [5000] 00000035 : 9A EB 86 26 2E AD 0B EC-EF FB EE AB B7 4F 74 BC : ...&.........Ot.
System.Net.Sockets Verbose: 0 : [5000] 00000045 : F3 31 E0 2A 0F 9F EB 00-35 00                   : .1.*....5.
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 74#74
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 16 03 01 03 AC                                  : .....
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
[server certificate received here. identical in both dumps]
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 940#940
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 16 03 01 00 09                                  : .....
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000005 : 0D 00 00 05 02 01 02 00-00                      : .........
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 9#9
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000000 : 16 03 01 00 04                                  : .....
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Receive()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Receive
System.Net.Sockets Verbose: 0 : [5000] 00000005 : 0E 00 00 00                                     : ....
System.Net.Sockets Verbose: 0 : [5000] Exiting Socket#29190913::Receive()   -> 4#4
System.Net.Sockets Verbose: 0 : [5000] Socket#29190913::Send()
System.Net.Sockets Verbose: 0 : [5000] Data from Socket#29190913::Send
[everything seems encrypted here]

and on the machine where the establishing SSL fails, that file begins with (dont mind the BeginConnectEx() as it seems like this is done by the .NET framework when calling from a thread in the ThreadPool):

System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Socket(InterNetwork#2)
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Socket() 
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::BeginConnect(142:3306#-1905378776)



System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::BeginConnect()  -> ConnectAsyncResult#45523402
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::EndConnect(ConnectAsyncResult#45523402)
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::EndConnect() 
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 3D 00 00 00 0A 35 2E 31-2E 35 34 2D 31 75 62 75 : =....5.1.54-1ubu
System.Net.Sockets Verbose: 0 : [3880] 00000010 : 6E 74 75 34 00 62 04 00-00 7A 27 51 43 3C 43 7B : ntu4.b...z'QC<C{
System.Net.Sockets Verbose: 0 : [3880] 00000020 : 64 00 FF FF 08 02 00 00-00 00 00 00 00 00 00 00 : d...............
System.Net.Sockets Verbose: 0 : [3880] 00000030 : 00 00 00 00 63 44 68 61-2E 5C 69 79 46 32 65 29 : ....cDha.\iyF2e)
System.Net.Sockets Verbose: 0 : [3880] 00000040 : 00                                              : .
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 65#65
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Send()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Send
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 04 00 00 01 8F AA 03 00-                        : ........
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Send()  -> 8#8
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Send()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Send
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 16 03 01 00 5A 01 00 00-56 03 01 4E 89 D1 92 4C : ....Z...V..N...L
System.Net.Sockets Verbose: 0 : [3880] 00000010 : 61 0C 93 9D 9F 7C 45 39-71 8C 0A 6E 29 7C D9 32 : a....|E9q..n)|.2
System.Net.Sockets Verbose: 0 : [3880] 00000020 : 89 CF 7B D8 1E 33 4F 90-A1 2C 27 00 00 18 00 2F : ..{..3O..,'..../
System.Net.Sockets Verbose: 0 : [3880] 00000030 : 00 35 00 05 00 0A C0 13-C0 14 C0 09 C0 0A 00 32 : .5.............2
System.Net.Sockets Verbose: 0 : [3880] 00000040 : 00 38 00 13 00 04 01 00-00 15 FF 01 00 01 00 00 : .8..............
System.Net.Sockets Verbose: 0 : [3880] 00000050 : 0A 00 06 00 04 00 17 00-18 00 0B 00 02 01 00    : ...............
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Send()  -> 95#95
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 16 03 01 00 4A                                  : ....J
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000005 : 02 00 00 46 03 01 81 9E-F1 4F F4 7A C6 21 B7 94 : ...F.....O.z.!..
System.Net.Sockets Verbose: 0 : [3880] 00000015 : 03 9B 37 C4 E0 EF 0D 1D-86 95 04 2B FE 31 6A A0 : ..7........+.1j.
System.Net.Sockets Verbose: 0 : [3880] 00000025 : DF 1A D8 32 82 9B 20 18-1A 7B C6 FC BB 89 0B DE : ...2.. ..{......
System.Net.Sockets Verbose: 0 : [3880] 00000035 : 97 71 EC 38 9E 14 29 4B-65 BF 23 17 6C 17 1C 3C : .q.8..)Ke.#.l..<
System.Net.Sockets Verbose: 0 : [3880] 00000045 : 6A 5E D9 13 BE CC F9 00-35 00                   : j^......5.
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 74#74
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 16 03 01 03 AC                                  : .....
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
[server certificate received here. identical in both dumps]
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 940#940
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 16 03 01 00 09                                  : .....
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000005 : 0D 00 00 05 02 01 02 00-00                      : .........
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 9#9
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000000 : 16 03 01 00 04                                  : .....
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 5#5
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Receive()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Receive
System.Net.Sockets Verbose: 0 : [3880] 00000005 : 0E 00 00 00                                     : ....
System.Net.Sockets Verbose: 0 : [3880] Exiting Socket#39785641::Receive()   -> 4#4
System.Net.Sockets Verbose: 0 : [3880] Socket#39785641::Send()
System.Net.Sockets Verbose: 0 : [3880] Data from Socket#39785641::Send
System.Net.Sockets Verbose: 0 : [3880] (printing 1024 out of 1540)
[my client certificate seems to be sent here]

(I've cut out the certificate parts of the files above). Anyway my guess is that the error that I receive on some machines is rooted in what cipher algorithm that is selected in the SSL handshake. But it's really a punch in the dark (?) and I'm tearing my hair at this very moment. The SSL handshake just seem to fail on some machines.

If anyone knows a good way to dissect/decode/analyze the traffic above (I've been in WireShark and banged my head for several hours, and still can't figure out how to give the above hex to the SSL dissector). Or if anyone can directly spot the difference / whats failing in #2 that would help alot!!

Answer 1

I found the solution myself while dissecting the TLS protocol. Turns out that the client that don't work in the example above is sending the my client certificate while handshaking; and the client that do work doesn't do that. Apparently encryption is established anyway (I haven't continued on into the TLS protocol), and probably a certificate exchange/key exchange is taken place further on.

To get the connection to work, all I had to do was change the connectionstring and remove all Certificate*= keys. In particular "Certificate Store Location=CurrentUser". My current, working MySql SSL connectionstring is:

server=xxx.yyy.zzz.uuu;database=whopper;user=Username;password=Secret;Pooling=false;SSL Mode=Required;Keepalive=60

Also, as a side note, while dissecting the communication, I found that Tamos CommView does a better job than WireShark in intercepting & dissecting while on VPN communications. Perhaps due to WinPCaps dissability to dissect VPN packets under Windows 7 x64. Also the TLS dissector in CommView is what really helped me find the handshaking issue.

Also, as a second side note. All SSL/TLS communication in Windows is handled by a DLL called schannel.dll. Full logging into the System EventLog for that dll can be enabled by creating the DWORD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\EventLogging with the value 7. Read more here: http://support.microsoft.com/kb/260729.

To make it work. Remove stuff.