How to debug a TLS implementation

Question

I am writing a TLS implementation and have been stuck at the server finished message for weeks now. Openssl s_client give sthe error:

5820:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:.\ssl \s3_pkt.c:483:

which is not very helpful because I cannot trace anything wrong to the MAC or the encryption process.
Chrome gives the error:

5820:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:.\ssl \s3_pkt.c:483:

while Firefox says:

Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap)

GNUtls-debug is even worse

error in the pull function.

I have narrowed down the suspect list to either a wrong des3/aes256 implementation(the only ones I've tried so far) or bad MAC, but I cannot be sure, and have been over the code for what feels like a thousand times with no luck of spotting anything evil going on.
Questions
Is there any way to get any browser to give a full and detailed SSL error? All I need is a detailed and reliable error message/debug log. Are there any tools for debugging a TLS implementation?

Answer 1

I'm in the same boat, updating an embedded TLS 1.0 implementation to TLS 1.2. I've found that Wireshark is great for troubleshooting as I go.

Go into the Wireshark Preferences to Protocols/SSL. From the RSA keys list you can give it a path to the private key (in my case, a .pem file) you're using on your server. Configure Wireshark with an SSL debug file, and it will dump extended debug information for each packet it decrypts.

Right now, I see that my AES256 decryption is broken since Wireshark decodes differently than my implementation. Earlier, I was getting the proper cleartext on an AES128 connection and had to work out some problems with my MAC calculations (or the finished messages, can't recall which). Without Wireshark, it was difficult to know whether the decryption or MAC steps had failed.