Address Bind exception in Tomcat

Question

We are running Java 8 web application in Apache Tomcat 8.0.36 server since 1 year. From past few days sometimes when we restart the Tomcat server , the application is not working. Logs are showing address bind exception for Https port. I tried checking the process associated with the port using below command in cmd:

netstat -aon | findstr [port]

but it is not showing any process associated with the port.

I have tried closing tomcat process from task manager and also tried executing tomact shutdown.bat and then started tomcat again but nothing helped.

If the system is restarted where application is deployed the issue gets resolved but we can't restart the system anytime

Please help what could be the issue and what else can be checked?

Below is the error in tomcat error log:

03-Aug-2018 11:42:32.008 INFO [main] org.apache.coyote.AbstractProtocol.init 
Initializing ProtocolHandler ["http-nio-443"]
03-Aug-2018 11:42:32.008 SEVERE [main] 
org.apache.coyote.AbstractProtocol.init Failed to initialize end point 
associated with ProtocolHandler ["http-nio-443"]
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:351)
at 
org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:748)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:459)
at 
org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11Jss
eProtocol.java:120)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:960)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardService.initInternal
(StandardService.java:568)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at  org.apache.catalina.core.StandardServer.initInternal
 (StandardServer.java:869)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.startup.Catalina.load(Catalina.java:580)
at org.apache.catalina.startup.Catalina.load(Catalina.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:310)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)

03-Aug-2018 11:42:32.008 SEVERE [main] 
 org.apache.catalina.core.StandardService.initInternal Failed to initialize 
 connector [Connector[HTTP/1.1-443]]
 org.apache.catalina.LifecycleException: Failed to initialize component 
[Connector[HTTP/1.1-443]]
 at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
 at org.apache.catalina.core.StandardService.initInternal
(StandardService.java:568)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal
(StandardServer.java:869)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.startup.Catalina.load(Catalina.java:580)
at org.apache.catalina.startup.Catalina.load(Catalina.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:310)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)
Caused by: org.apache.catalina.LifecycleException: Protocol handler 
initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:962)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
... 12 more
Caused by: java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:351)
at 
org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:748)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:459)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init
(AbstractHttp11JsseProtocol.java:120)
 at org.apache.catalina.connector.Connector.initInternal(Connector.java:960)
 ... 13 more03-Aug-2018 11:42:32.008 INFO [main] 
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http- 
nio-443"]
03-Aug-2018 11:42:32.008 SEVERE [main] 
org.apache.coyote.AbstractProtocol.init 
Failed to initialize end point associated with ProtocolHandler ["http-nio- 
443"]
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:351)
at org.apache.tomcat.util.net.AbstractEndpoint.init
(AbstractEndpoint.java:748)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:459)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init
(AbstractHttp11JsseProto col.java:120)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:960)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardService.initInternal
(StandardService.java:568)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal
(StandardServer.java:869)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.startup.Catalina.load(Catalina.java:580)
at org.apache.catalina.startup.Catalina.load(Catalina.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:310)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)

03-Aug-2018 11:42:32.008 SEVERE [main] 
org.apache.catalina.core.StandardService.initInternal Failed to initialize 
connector [Connector[HTTP/1.1-443]]
org.apache.catalina.LifecycleException: Failed to initialize component 
[Connector[HTTP/1.1-443]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
at org.apache.catalina.core.StandardService.initInternal
(StandardService.java:568)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal
(StandardServer.java:869)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.startup.Catalina.load(Catalina.java:580)
at org.apache.catalina.startup.Catalina.load(Catalina.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:310)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:484)
Caused by: org.apache.catalina.LifecycleException: Protocol handler 
initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:962)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
... 12 more
Caused by: java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:351)
at org.apache.tomcat.util.net.AbstractEndpoint.init
(AbstractEndpoint.java:748)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:459)
at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init
(AbstractHttp11JsseProtocol.java:120)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:960)
... 13 more

Answer 1

See the update below. We think we have found the root cause and a solution

We have the same problem: A tomcat service that was operating without problems for nearly a year was not able to bind to the port which it used after it was stopped and then restarted.

We usually stop and restart our tomcat service regularly every night, and it used to work without problems. The problem first occurred on July 24th, so after a lot of research that I and our IT have done, I suspect that some kind of change in our IT infrastructure that happened shortly before July 24th causes this problem.

Our server is a Windows Server 2012 R2, 64 Bit, but our IT environment is virtualized, running on vSphere Host Version 6. On what server/infrastructure is your tomcat operating? Is it a virtualized system or is it running on a physical server?

In my opinion, the problem came in either with the last installed patches for Windows (KB4338815 and/or KB4339093) or with some Hypervisor updates. But our IT is still researching.

For the moment as a workaround we do a server restart when we shutdown the service and restart it regularly every night. But because the problem does not reliable occur on each and every shutdown/restart, we restart the server only if it is really necessary. To achieve this we use a listener we have implemented and configured in the server.xml that scans the stderr.log just after the tomcat service has started. If it detects, that a Bind Exception occurs it restarts the server (tomcat service is installed as automatic). If you find the cause of this problem, please post it here, as I will, if I or our IT will find it.

Update August 7th: We think we have found the root cause. Obviously the Windows Monthly Quality Rollup for July introduced this problem. In our case (Windows Server 2012 R2, 64 Bit) the problematic update was KB4338815 (see https://support.microsoft.com/en-us/help/4338815/windows-81-update-kb4338815).

If you do a quick research on the internet, the problem, which also affects Windows IIS Webserver and MS SQL Server is mostly described as related to KB4338818 (see https://support.microsoft.com/en-us/help/4338818/windows-7-update-kb4338818) but this is the same kind of update for Windows Server 2008 R2 and Windows 7.

Microsoft is already aware of the problem and describes it in its documentation for KB4338815 and KB4338818 in the "Known issues in this update" section of the mentioned articles. Microsoft states that the problem is resolved in KB4338831 (for Windows Server 2012 R2, Windows 8.1) and KB4338821 (for Windows Server 2008 R2, Windows 7)

In our case we installed hot fix KB4345424 (see https://support.microsoft.com/en-us/help/4345424/improvements-and-fixes-windows-8-1-and-server-2012-r2), also available from Microsoft, which addresses this bug and as far as we see, the problem does not exist anymore.