Unable to deploy CXF3.x JAX-RS service in Weblogic 12c (12.2.1)

Question

After upgrading a JAX-RS service built with CXF3.x from weblogic 12.1.3 to 12.2.1 I am facing the following weird issue

<03-ago-2017, 3:22:38,789 PM CEST> <Error> <HTTP> <BEA-101216> <Servlet: "JAX-RS/Jersey#1" failed to preload on startup in Web application: "sife".
A MultiException has 1 exceptions.  They are:
1. java.lang.LinkageError: ClassCastException: attempting to castzip:C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/sife/845176/war/WEB-INF/lib/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/C:/Oracle/Middleware/Oracle_Home/oracle_common/modules/javax.ws.rs.javax.ws.rs-api.jar!/javax/ws/rs/ext/RuntimeDelegate.class

        at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:1007)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:962)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:1054)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:1046)
        at org.glassfish.jersey.model.internal.CommonConfig.configureFeatures(CommonConfig.java:696)
        Truncated. see log file for complete stacktrace

Note that I am using CXF, not Jersey, but seems Weblogic is trying to use it (it does not happen with 12.1.3). I disabled it in configuration files following this recommendations: https://jerometambo.github.io/blog/2016/12/13/How-to-use-CXF-3x-implementation-of-JAX-RS-20-REST-with-Weblogic-12c.html

With default configuration, Weblogic wants to use its internal implementation (JAX-RS 2.0 and serialization) to deploy the REST web services (Error 500):

META-INF/weblogic-application.xml

<weblogic-application  xmlns:wls="http://xmlns.oracle.com/weblogic/weblogic-application"
                       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                       xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-application
                                           http://xmlns.oracle.com/weblogic/weblogic-application/1.8/weblogic-application.xsd">

    <prefer-application-packages>
        <package-name>javax.ws.rs.*</package-name>
        <package-name>com.fasterxml.jackson.*</package-name>
        <package-name>weblogic.jaxrs.api.client.*</package-name>
        <package-name>weblogic.jaxrs.internal.api.*</package-name>
        <package-name>weblogic.jaxrs.dispatch.*</package-name>
        <package-name>weblogic.jaxrs.monitoring.util.*</package-name>
    </prefer-application-packages>
</weblogic-application>

WEB-INF/weblogic.xml

<weblogic-web-app 
    xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd">
        
    <container-descriptor>
        <prefer-application-packages>
            <package-name>javax.ws.rs.*</package-name>
            <package-name>com.fasterxml.jackson.*</package-name> 
        </prefer-application-packages>
    </container-descriptor> 
</weblogic-web-app>

I do not know if this is a specific problem of my project or it is a weblogic issue. I have found similar issues like this and this, and I tried to use them.

These are my dependencies (IVY syntax)

<dependency org="org.apache.cxf" name="cxf-rt-frontend-jaxrs" rev="3.1.10" conf="default" />
    <dependency org="org.apache.cxf" name="cxf-rt-rs-extension-providers" rev="3.1.10" conf="default" />
    <dependency org="org.apache.cxf" name="cxf-rt-transports-http" rev="3.1.10" conf="default" />
    <dependency org="com.fasterxml.jackson.jaxrs" name="jackson-jaxrs-json-provider" rev="2.6.6" conf="default" />

And the full stack trace

<03-ago-2017, 3:32:03,645 PM CEST> <Warning> <JAXRSIntegration> <BEA-2192510> <Cannot add Jersey servlet for application class org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet$ApplicationImpl because ApplicationPath annotation is not set on it.>
<03-ago-2017, 3:32:03,737 PM CEST> <Warning> <JAXRSIntegration> <BEA-2192511> <The list of resource packages: com.fasterxml.jackson.jaxrs.base;org.apache.cxf.jaxrs.provider;com.fasterxml.jackson.jaxrs.json;org.apache.cxf.jaxrs.provider.aegis;es.aragon.sife.server.filter;org.apache.cxf.jaxrs.provider.jsrjsonp;es.aragon.sife.keystore.api;es.aragon.sife.server.account.api;org.apache.cxf.jaxrs.provider.xmlbeans;org.apache.cxf.jaxrs.validation;es.aragon.sife.server.task.api;org.apache.cxf.jaxrs.provider.json;org.apache.cxf.jaxrs.provider.atom;es.aragon.sife.server.config.api>
ago 03, 2017 3:32:16 PM es.gob.afirma.signers.xml.Utils installXmlDSigProvider
<03-ago-2017, 3:32:20,581 PM CEST> <Error> <HTTP> <BEA-101216> <Servlet: "JAX-RS/Jersey#1" failed to preload on startup in Web application: "sife".
A MultiException has 1 exceptions.  They are:
1. java.lang.LinkageError: ClassCastException: attempting to castzip:C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/sife/845176/war/WEB-INF/lib/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/C:/Oracle/Middleware/Oracle_Home/oracle_common/modules/javax.ws.rs.javax.ws.rs-api.jar!/javax/ws/rs/ext/RuntimeDelegate.class

        at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:1007)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:962)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:1054)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:1046)
        at org.glassfish.jersey.model.internal.CommonConfig.configureFeatures(CommonConfig.java:696)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.LinkageError: ClassCastException: attempting to castzip:C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/sife/845176/war/WEB-INF/lib/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/C:/Oracle/Middleware/Oracle_Home/oracle_common/modules/javax.ws.rs.javax.ws.rs-api.jar!/javax/ws/rs/ext/RuntimeDelegate.class
        at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:146)
        at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:120)
        at javax.ws.rs.core.MediaType.valueOf(MediaType.java:179)
        at org.glassfish.jersey.media.sse.SseFeature.<clinit>(SseFeature.java:62)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        Truncated. see log file for complete stacktrace
>
<03-ago-2017, 3:32:20,696 PM CEST> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application "sife".
weblogic.application.ModuleException: java.lang.LinkageError: ClassCastException: attempting to castzip:C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/sife/845176/war/WEB-INF/lib/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/C:/Oracle/Middleware/Oracle_Home/oracle_common/modules/javax.ws.rs.javax.ws.rs-api.jar!/javax/ws/rs/ext/RuntimeDelegate.class
        at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:140)
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:233)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:228)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.LinkageError: ClassCastException: attempting to castzip:C:/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/servers/AdminServer/tmp/_WL_user/sife/845176/war/WEB-INF/lib/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/C:/Oracle/Middleware/Oracle_Home/oracle_common/modules/javax.ws.rs.javax.ws.rs-api.jar!/javax/ws/rs/ext/RuntimeDelegate.class
        at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:146)
        at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:120)
        at javax.ws.rs.core.MediaType.valueOf(MediaType.java:179)
        at org.glassfish.jersey.media.sse.SseFeature.<clinit>(SseFeature.java:62)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        Truncated. see log file for complete stacktrace
>
<03-ago-2017, 3:32:20,762 PM CEST> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>
<03-ago-2017, 3:32:20,931 PM CEST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>
<03-ago-2017, 3:32:21,46 PM CEST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>
<03-ago-2017, 3:32:21,174 PM CEST> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>
<03-ago-2017, 3:32:21,186 PM CEST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>
<03-ago-2017, 3:32:22,710 PM CEST> <Error> <HTTP> <BEA-101216> <Servlet: "weblogic.management.rest.Application" failed to preload on startup in Web application: "wls-management-services.war".
A MultiException has 1 exceptions.  They are:
1. java.lang.NoClassDefFoundError: Could not initialize class org.glassfish.jersey.media.sse.SseFeature

        at org.jvnet.hk2.internal.Utilities.justCreate(Utilities.java:1007)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.create(ServiceLocatorImpl.java:962)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:1054)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.createAndInitialize(ServiceLocatorImpl.java:1046)
        at org.glassfish.jersey.model.internal.CommonConfig.configureFeatures(CommonConfig.java:696)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.NoClassDefFoundError: Could not initialize class org.glassfish.jersey.media.sse.SseFeature
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1350)
        Truncated. see log file for complete stacktrace
>
<03-ago-2017, 3:32:22,747 PM CEST> <Error> <J2EE> <BEA-160001> <Error deploying Internal Application: weblogic.application.ModuleException: java.lang.NoClassDefFoundError: Could not initialize class org.glassfish.jersey.media.sse.SseFeature
weblogic.application.utils.StateChangeException: weblogic.application.ModuleException: java.lang.NoClassDefFoundError: Could not initialize class org.glassfish.jersey.media.sse.SseFeature
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:58)
        at weblogic.application.services.BackgroundDeploymentManagerService$BackgroundDeployAction.run(BackgroundDeploymentManagerService.java:396)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:666)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)
        Truncated. see log file for complete stacktrace
Caused By: weblogic.application.ModuleException: java.lang.NoClassDefFoundError: Could not initialize class org.glassfish.jersey.media.sse.SseFeature
        at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:140)
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:233)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:228)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.NoClassDefFoundError: Could not initialize class org.glassfish.jersey.media.sse.SseFeature
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1350)
        Truncated. see log file for complete stacktrace

---

WORKAROUND

It works if I deploy previously other CXF module using the default JAX-RS implementation and removing WEB-INF/lib/javax.ws.rs-api-2.0.1.jar. Maybe weblogic does some kind of internal initialization.

 myapp.ear
   init.war (JAX-RS with CXF and removing javax.ws.rs-api-2.0.1.jar)
   myapp.war (JAX-RS with CXF and weblogic configuration)

Answer 1

@pedrofb, this may not fully answer your question, however, I once had a similar issue with Tomcat. The symptoms are similar, the problem was not really due to the presense of different class present in different jars i.e. jar shadowing. It was due to classloader shadowing. In Tomcat, classloading is actually inverted, not the usual 'ask-your-parent-classloader-first' kind of model, if an application classloader can find a class (where it only loads from WEB-INF/lib or classes) then the classloader loads the class. Now, in your case, I think, since the JAX-RS jars are present in your WEB-INF/lib, it is getting loaded from there at web application level. However, when the same object is being passed to the container, it is trying to cast it to the same class but at container level loaded by a different classloader. Classes loaded by different classloaders are considered different; this would be the cause for ClassCastException.

This would also explain why there is no ClassCastException when you remove the JAX-RS jar from WEB-INF/lib. The workaround here are limited, either you remove the JAX-RS jar from your WEb-INF/lib as you have done. (This could be potentially problematic but you could experiment in non production server), replace the container JAX-RS jar with the one that you want.

As I said, this will probably not solve your problem completely but may point in a direction to where the problem is coming from. [As I noticed some years back, Weblogic uses tomcat as its web container core (not sure if it is still the same) with changed package names like com.bea.weblogic.org.apache.tomcat ...].

Hope this helps.