javax.xml.ws.WebServiceException: java.io.IOException: Error writing to server Tomcat 8

Question

We have recently updated the tech stack of a JAX-WS webservice running on JRE 1.7.0_17/Tomcat7.0.39 to JRE 1.8.0_66/Tomcat 8.0.28. The web app runs on Windows Server 2012. The web service uses a Metro implementation for JAX-WS. The clients run on various windows versions using JRE 7 and the JAX-WS client API built into the JRE. The webservice is used to upload files from the client machines to the webservice which saves them in a document management system. The implementation worked pretty much flawlessly under Java 7/Tomcat 7 but we have run into a problem with larger payloads (2MB or larger) running under Java 8/Tomcat 8 server side. The stack trace from the client is:

12/02/2015 14:12:38.699 [AWT-EventQueue-0] ERROR  DocumentImporterMainWindow$SwingAction.importDocument: Unexpected Problem trying to call the CustomerOrderDMService
javax.xml.ws.WebServiceException: java.io.IOException: Error writing to server
    at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(Unknown Source)
    at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.createResponsePacket(Unknown Source)
    at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown Source)
    at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown Source)
    at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown Source)
    at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source)
    at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source)
    at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source)
    at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source)
    at com.sun.xml.internal.ws.client.Stub.process(Unknown Source)
    at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source)
    at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
    at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
    at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
    at com.sun.proxy.$Proxy30.importDocument(Unknown Source)
    at com.mycompany.documentimporter.DocumentImporterMainWindow$SwingAction.importDocument(DocumentImporterMainWindow.java:681)
    at com.mycompany.documentimporter.DocumentImporterMainWindow$SwingAction.actionPerformed(DocumentImporterMainWindow.java:612)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
    at java.awt.AWTEventMulticaster.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Window.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
    at java.awt.EventQueue.access$500(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue.dispatchEvent(Unknown Source)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.run(Unknown Source)
Caused by: java.io.IOException: Error writing to server
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.net.HttpURLConnection.getResponseCode(Unknown Source)
    ... 54 more

Unfortunately nothing is logged server side in any of the tomcat logs. I have spent days searching for a solution to the problem without success. I have tried to debug the problem through various avenues such as logging the SOAP request/response client side and server side using the -Dcom.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true (client) and -Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true(server) java system properties but when the error occurs only the Client side request is being logged:

---[HTTP request - http://localhost:8080/CustomerOrderDM/services/CustomerOrderDMService]---
Accept: text/xml, multipart/related
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://www.mycompany.com/CustomerOrderDMService/ImportDocument"
User-Agent: JAX-WS RI 2.2.9-b130926.1035 svn-revision#5f6196f2b90e9460065a4c2f4e30e065b245e51e
<?xml version="1.0" ?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns3:ImportDocumentRequestDTO xmlns:ns3="http://www.mycompany.com/CustomerOrderDMService/" xmlns:ns2="http://www.mycompany.com/CustomerOrderDMService"><ns2:QuoteNumber>A000049</ns2:QuoteNumber><ns2:SerialNumber>STOCK</ns2:SerialNumber><ns2:DocumentType>Email</ns2:DocumentType><ns2:Description></ns2:Description><ns2:DocumentContents> **some base64 encoded byte[] of the file contents being uploaded**

Message has been truncated
use com.sun.xml.internal.ws.transport.http.HttpAdapter.dumpTreshold property to increase the amount of printed part of the message
--------------------

Since only the client side request is being logged we anticipate the server is not completely processing the request and it is falling into some kind of exception block, but without anything logging to the server log files we are having difficulties trouble shooting the problem.

We have tried using proxies such as the Monitoring built into Eclipse, but once again I only see the request from the client and no response from the server (when the client sends the larger requests that fail, small requests log request/response on both client and server). Other suggestions for debugging would be greatly appreciated.

We have also tried different combinations of Java and Tomcat:

• Tomcat 7/Java 7 = Works
• Tomcat 7/Java 8 = Works
• Tomcat 8/Java 7 = Doesn't Work
• Tomcat 8/Java 8 = Doesn't Work

This leads us to think that the issue is with Tomcat 8. Either something was changed in Tomcat 8 and we now need to set some new timeout/payload settings or Tomcat 8 has a bug related to this specific problem.

We have tried setting some of the Tomcat connector settings like maxPostSize="-1", connectionTimeout="-1", disableUploadTimeout="true", connectionUploadTimeout="-1", keepAliveTimeout="-1" but none of these worked and honestly feel like a shot in the dark without knowing what is going on server side.

We have tried updating the Metro jars server side to the most recent release (jaxws-ri-2.2.10) as well as running the client using Java 8. Unfortunately none of these worked. Any help would be greatly appreciated.

Answer 1

It turns out that Tomcat 7.0.55 included Fix:

CVE-2014-0230:
Add a new limit, defaulting to 2MB, for the amount of data Tomcat will swallow for an aborted upload. The limit is configurable by maxSwallowSize attribute of an HTTP connector. 

My issues was solved by setting the maxSwallowSize="-1" setting on the <Connector> configuration in the tomcat servers server.xml.

I would like to thank Mark Thomas and Chris Schultz from the Tomcat users mailing list for their help. For instructions on how to join the mailing list click here. I would also like to thank Vinayak for pointing me towards Tomcat support.

Answer 2

Seems this is more a bug related to Tomcat. Post the issue to Tomcat team.

http://tomcat.apache.org/bugreport.html