SSL_ERROR_RX_RECORD_TOO_LONG with custom server

Question

I've created a web server based on sockets. It can serve files via HTTPS or HTTP.

The issue I'm experiencing occurs when requesting a file, such as an image, from the server via HTTPS using Firefox and Chromium.

When Firefox requests the image from the server at https://localhost:8443/ada.jpg, Firefox reports

SSL_ERROR_RX_RECORD_TOO_LONG

Similarly, Chromium errors with

ERR_SSL_PROTOCOL_ERROR

The following works fine though:

• cURL getting the image via HTTPS: curl -Ok "https://localhost:8443/ada.jpg"
• Getting a text response via HTTPS using cURL, Firefox, and Chromium: https://localhost:8443/
• Getting a smaller image (6773 bytes) like this one via HTTPS using cURL, Firefox, and Chromium
• Getting the image via HTTP (no TLS) using cURL, Firefox, and Chromium: http://localhost:8443/ada.jpg

I've used Wireshark to look at the frames while the browsers are getting the image: Both Firefox and Chromium send a [FIN, ACK] frame to the server while the image is still being transmitted. The server continues sending parts of the image after which the browsers send a [RST] frame.

These are the last couple of frames from the exchange between Firefox and the server:

25  1.873102771 ::1 ::1 TCP 86  55444 → 8443 [ACK] Seq=937 Ack=18043 Win=56704 Len=0 TSval=3976879013 TSecr=3976879013
26  1.873237965 ::1 ::1 TLSv1.3 110 Application Data
27  1.873247272 ::1 ::1 TCP 86  8443 → 55444 [ACK] Seq=18043 Ack=961 Win=65536 Len=0 TSval=3976879013 TSecr=3976879013
28  1.873346910 ::1 ::1 TCP 86  55444 → 8443 [FIN, ACK] Seq=961 Ack=18043 Win=65536 Len=0 TSval=3976879013 TSecr=3976879013
29  1.876736432 ::1 ::1 TLSv1.3 16508   Application Data
30  1.876769660 ::1 ::1 TCP 74  55444 → 8443 [RST] Seq=962 Win=0 Len=0

Here are the Wireshark captures of cURL and Firefox.

---

For reproducing the error, do

git clone [email protected]:bullbytes/simple_socket_based_server.git
cd simple_socket_based_server
./gradlew run

This starts the server with TLS enabled. Then, open the following URL in your browser (the first time you'll have to add an exception for the self-signed certificate):

https://localhost:8443/ada.jpg

To convince yourself that transferring the image succeeds when TLS is disabled, start the server like this:

./gradlew run --args="--use-tls=no"

After removing the s from the protocol, the image should show up in your browser:

http://localhost:8443/ada.jpg

The server's request-handling loop is here.

How can I debug this further and fix it?

---

I'm on Arch Linux 5.2.9 using OpenJDK 12.0.2+10.

Answer 1

This is a bug in the JDK

It was fixed¹ in JDK 13.

The original bug report is here with the title

TLSv1.3 may generate TLSInnerPlainText longer than 2^14+1 bytes

which is consistent with the observations in the question: A record longer than that violates the TLSv1.3 protocol.

This bug report states that Java's TLS implementation is fixed in JDKs 11.0.5 b01 and 13 via this commit.

Generally, JDK 13 reimplemented Socket and ServerSocket², making them play well with fibers introduced in project Loom.

The bug I filed for JDK 12 is here.

Workaround

If upgrading the JDK is not possible in your situation, you could use ServerSocketChannel instead of ServerSocket for now. Jetty does that.

---

¹_{The bug does not occur with OpenJDK build 13+33 on Arch Linux: The image shows up fine in Firefox}

²_{The server used to reproduce the error is built on ServerSocket.}