WebSphere 9 excessive startup time

Question

I'm test driving WebSphere Application Server 9, full profile, and I'm intrigued about the time it takes to startup the profile. I already installed an application and, in theory, this could be the culprit, but the logs do not show any evidence of that. The relevant portion of the log file:

[14-07-2016 9:15:06:955 BST] 0000005b ApplicationMg A   WSVR0221I: Application started: isclite
[14-07-2016 9:15:06:956 BST] 0000005b CompositionUn A   WSVR0191I: Composition unit WebSphere:cuname=isclite in BLA WebSphere:blaname=isclite started.
[14-07-2016 9:16:14:410 BST] 0000005a InternalGener I   DSRA8225I: DataSource JNDI name : jdbc/DefaultEJBTimerDataSource
[14-07-2016 9:16:14:413 BST] 0000005a InternalGener I   DSRA8203I: Database product name : Apache Derby
[14-07-2016 9:16:14:413 BST] 0000005a InternalGener I   DSRA8204I: Database product version : 10.11.1.1 - (1616546)
[14-07-2016 9:16:14:413 BST] 0000005a InternalGener I   DSRA8205I: JDBC driver name  : Apache Derby Embedded JDBC Driver
[14-07-2016 9:16:14:414 BST] 0000005a InternalGener I   DSRA8206I: JDBC driver version  : 10.11.1.1 - (1616546)
[14-07-2016 9:16:14:414 BST] 0000005a InternalGener I   DSRA8218I: JDBC driver specification level  : 4.2
[14-07-2016 9:16:14:773 BST] 0000005a WASSchedulerC I   SCHD0100I: Scheduler tables verified successfully.

As you can see on the 2nd and 3rd lines, the server waits for more than a minute. There are no exceptions or any other information being logged that might indicate a problem.

Does anyone know what could be the problem?

EDIT #1:

Following XSurgent's suggestion, I've disabled my application's auto start and restarted the server. The startup time was normal, but the lines indicating that Apache Derby is starting are no longer visible, which seems to indicate that it is Derby startup that is consuming all that time. Does my reasoning make any sense?

EDIT #2:

I've changed the EJB Timer to use a different database (Oracle instead of the embedded Derby), but the 1 minute delay is still happening.

Answer 1

Application startup times can increase for applications which contain large web modules. ("Large" meaning has many classes, typically because many JAR files are packaged in the web module.) One cause of the increase are new scans of class information to detect CDI content. For example, to detect @Inject annotations.

The scans are required -- even for modules which contain no CDI content -- for specification compliance for the new CDI 1.2 feature. The scans have a cost in proportion to the number of classes in modules which are scanned.

To improve startup performance, two properties were introduced to disable CDI processing for applications which do not contain CDI content. See this technical note for a description of the properties and usage instructions:

http://www-01.ibm.com/support/docview.wss?uid=swg21983564

The two properties are:

As a java custom property: "com.ibm.ws.cdi.enableImplicitBeanArchives"
As a filter property: "Enable-Implicit-Bean-Archive"  

As a java custom property: "com.ibm.ws.cdi.enableCDI"
As a filter property: "Enable-CDI"

Here "java custom property" means a property set in the server process. Note that this must be done in the server configuration: Setting the property in the launch script will only affect the launch process, which is not the server process.

"filter property" means a property which can be set in the file "amm.filter.properties" or a manifest main attribute which can be set in a Java EE archive (EAR/WAR/RAR/JAR file). (More detailed instructions for setting filter properties are provided in the technical note.)

Here is a summary of the properties and their meanings:

Name: com.ibm.ws.cdi.enableImplicitBeanArchives, Enable-Implicit-Bean-Archive
Allowed Values: true, false
Default Value: true
Description: By setting the property to false, CDI-related annotation scanning steps are disabled for archives that do not contain a bean descriptor (beans.xml).

Name: com.ibm.ws.cdi.enableCDI, Enable-CDI
Allowed Values: true, false
Default Value: true
Description: By setting the property to false, all CDI steps are disabled for the archive.

Which property to use depends on the specific requirements: Set the java custom property if all CDI function is to be disabled for the server process. Set manifest main attributes to disable CDI function for specific applications or modules.

A way to tell if CDI is causing the increased startup times is to collect thread dumps and look for call stacks similar to the following:

4XESTACKTRACE                at com/ibm/ws/jsf/ext/JSFInjectionClassListCollaborator.getInjectionClasses(JSFInjectionClassListCollaborator.java:105)
4XESTACKTRACE                at com/ibm/ws/cdi/classic/CDIArchiveImpl.getInjectionClassList(CDIArchiveImpl.java:77)
4XESTACKTRACE                at com/ibm/ws/cdi/impl/weld/BeanDeploymentArchiveImpl.initializeJEEComponentClasses(BeanDeploymentArchiveImpl.java:349)
4XESTACKTRACE                at com/ibm/ws/cdi/impl/weld/BeanDeploymentArchiveImpl.scan(BeanDeploymentArchiveImpl.java:281)
4XESTACKTRACE                at com/ibm/ws/cdi/impl/weld/BeanDeploymentArchiveImpl.scan(BeanDeploymentArchiveImpl.java:255)
4XESTACKTRACE                at com/ibm/ws/cdi/impl/weld/WebSphereCDIDeploymentImpl.scan(WebSphereCDIDeploymentImpl.java:530)
4XESTACKTRACE                at com/ibm/ws/cdi/impl/CDIContainerImpl.applicationStarting(CDIContainerImpl.java:117)
4XESTACKTRACE                at com/ibm/ws/cdi/classic/CDIRuntimeImpl.applicationStarting(CDIRuntimeImpl.java:323)

Thomas Bitonti, IBM Corporation, Developer for IBM WebSphere Application Server