CloseNowException: This stream is not writeable

Question

I'm really stumped. In my server logs I'm seeing:

org.apache.coyote.CloseNowException: Connection [215], Stream [95], This stream is not writable
        at org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:843) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.http2.Stream$StreamOutputBuffer.doWrite(Stream.java:859) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.http2.Http2OutputBuffer.doWrite(Http2OutputBuffer.java:59) ~[tomcat-coyote.jar:9.0.30]
        at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-coyote.jar:9.0.30]

It seems to happen when a user clicks too quickly, but that makes no sense, as Tomcat should be able to serve plenty of requests. This is on a server with a very light load, maybe 2 or 3 HTTP requests per second, on a very fast machine.

This is with Spring Boot 2 and Tomcat 9.0.30. It's really perplexing.

I did see a similar question on SO where someone got this using a web push, but we're not.

Here's how our HTTP/2 connector is configured:

<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
           maxThreads="150" SSLEnabled="true"
           keystoreFile="/etc/ssl/keys.p12"
           keystorePass="changeit"
           keyAlias="tomcat"
           sslProtocol="TLS"
           sslEnabledProtocols="TLSv1.3,TLSv1.2"
           connectionTimeout="20000"
           >
  <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
                   keepAliveTimeout="20000"
                   />
</Connector>

It's running on JDK 13.0.2 on Ubuntu server 18.04.

Any ideas on this? It's definitely something users notice and I have no clue about how to solve this.

Answer 1

This problem actually didn't go away. I tried many many things. I tried this on several newer version of Tomcat in the 9.0 series. I upgraded to JDK 14, 14.0.1 and 14.0.2, all from Oracle (I never tried the OpenJDK release, can't imagine it would make any difference). I tried all kinds of TLS settings, including limiting it to only TLS1.3.

The problem continued.

I tried running a completely different Spring Boot application, one that had to make a large number of small (500 byte) JSON POSTs to the server. Very often these POST messages would fail. It happened regularly on some browsers but not others. There wasn't any clear pattern to this. These would be browsers running on Windows, Linux and Safari, and on mobile devices. Sometimes it worked sometimes not.

Very often the most likely to fail GETs would be getting small static objects, like a small CSS file, which should in fact be the EASIEST and fastest thing to load. As a side consequence, failing to load CSS totally breaks the site, so this is really bad.

So we have many different versions of Tomcat, many versions of JDK, many different variations of config parameters for http2 and TLS, and two completely different Spring Boot applications, and we get the same problem: sporadic failures of http2 sessions. The only thing in common here is that the problems all go away when I switched back to http1.1.

Http1.1 as reliable as it is has significantly worse performance than http2 and I wanted it to work.

Out of total desperation I thought, ok, the only thing I haven't changed is the server software, and I know that Jetty's http2 implementation is completely different from Tomcat's.

I switched to a basic Jetty configuration, enabled http2, got my app deployed, and tested it. All the http2 problems went away. It started behaving as expected, performing well, and had the performance benefits of http2. This switch fixed the problems on both the Spring Boot apps that had problems before. By the way these two Spring Boot apps were very different, one involving a database and Spring Integration and tons of features, the other much simpler, involving reading and writing a few small files.

I can only reach one conclusion here: something is wrong in Tomcat's http2 implementation and it has not had sufficient testing or real-world use and should not be used in production. This was a painful thing to learn as I've been using Tomcat as my go-to server for many years and I assumed Tomcat releases would be stable and production ready but this simply was not. If you are running into these "closed stream" errors where the client simply fails to load, and it's immediate, not after any timeout, and it happens with http2 but not http1, then try switching to Jetty and see if that makes a difference. If any Tomcat developers wants to get further into this, reply to this and I can set up our system as a demo.

I really came to this with a strong bias in favor of Tomcat, given that it's the oldest and most widely used Servlet container, but by this point I conclude that Jetty is going to be my go-to.

I hope this post helps others who may be encountering this problem because it really was not easy to figure out, and in the end I didn't ever figure out what was happening, other than, something is amiss in Tomcat's http2.

Answer 2

The solution is very simple. I have had the same problem.

There are other parameter to configure in in the following link there is a detailed list https://tomcat.apache.org/tomcat-8.5-doc/config/http2.html

In your case a simple solution is :

<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"                     
                 overheadCountFactor="-1"
                 overheadDataThreshold="0"
                 overheadWindowUpdateThreshold="0"/>

overheadCountFactor: The factor to apply when counting overhead frames to determine if a connection has too high an overhead and should be closed. The overhead count starts at -10. The count is decreased for each data frame sent or received and each headers frame received.ecc ecc ...... if the overhead count exceeds zero, the connection is closed. A value of less than 1 disables this protection. In normal usage a value of 3 or more will close the connection before any streams can complete. If not specified, a default value of 1 will be used.