OpenShift Tomcat Processing Annotations Error Deploying War

Question

I updated a java war on open shift and after the update the app won't start. When I looked at the jbossews log file I found the issue but I can't duplicate it outside OpenShift. It's woking fine on tomcat locally.

I get a series of processing errors until the app runs out of heap space:

May 22, 2015 12:04:16 AM org.apache.catalina.startup.ContextConfig processAnnotationsJar
SEVERE: Unable to process Jar entry [javassist/ByteArrayClassPath.class] from Jar [jar:jndi:/localhost/cope/WEB-INF/lib/javassist-3.19.0-GA.jar!/] for annotations
java.io.EOFException
    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
    at org.apache.tomcat.util.bcel.classfile.Utility.swallowMethodParameters(Utility.java:797)
    at org.apache.tomcat.util.bcel.classfile.Attribute.readAttribute(Attribute.java:171)
    at org.apache.tomcat.util.bcel.classfile.FieldOrMethod.<init>(FieldOrMethod.java:57)
    at org.apache.tomcat.util.bcel.classfile.Method.<init>(Method.java:71)
    at org.apache.tomcat.util.bcel.classfile.ClassParser.readMethods(ClassParser.java:267)
    at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:127)
    at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2058)
    at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1934)
    at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1900)
    at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1885)
    at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1317)
    at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:876)
    at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:374)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5355)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1083)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1880)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

This goes on until the following:

May 22, 2015 12:04:17 AM org.apache.catalina.startup.HostConfig deployWARs
SEVERE: Error waiting for multi-thread deployment of WAR files to complete
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:188)
    at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:818)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1614)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:330)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
    at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:402)
    at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:347)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1140)
    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:799)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space

What am I missing here??

Answer 1

So, to answer my own question, the issue appears to be with the latest version of javassist. We updated to the latest version as part of the last update, 3.19.0, and it looks like it's not compatible with Tomcat on OpenShift.Once I downgraded back to 3.17.1 all worked fine.

Just as a note to any OpenShift staff, it would really help if issues like this can be highlighted on the OpenShift support pages. Javassist 3.19.0 has been around for 5 months on maven and I can't imagine we were the first to encounter this issue.

Answer 2

This problem is confirmed fixed in the Tomcat 7.0.56 changelog.

See this question for more info: Tomcat 7 Annotation scanner and JDK 8 with -parameters compiler flag exceptions

Javassist 3.19.0 used java8 compiler argument -parameters