Websphere Application Server - What on earth will it take to start any fast?

Question

I am using Rational Application Developer v7.0 that ships with an integrated test environment. When I get to debugging my webapp, the server startup time in debug mode is close to 5-6 minutes - enough time to take a coffee break!

At times, it so pisses me off that I start cursing IBM for building an operating system instead of an app server! Spawning 20+ processes and useless services with no documented configuration to tuning it, to starting any faster.

I am sure there are many java developers out there who would agree with me on this. I tried to disable the default apps and a set of services via my admin console, however that hasn't helped much.

I have no webservices, no enterprise beans, no queues, just a simple web app which requires a connection pool. Have you done something in the past to make your integrated test environment, start fast in debug mode and there by consume less RAM?

UPDATE: I tried disabling a few services (internationalization, default apps etc...) and now the WebSphere server went from bad to worse. Not only doesn't it take horrifying startup time, it keeps freezing every now and then for up to 2 minutes. :-( Sounds like, optimization is not such a good thing, always!

Answer 1

The best way to debug server code is to use remote debugging.

First you need to add the following to the JVM params in the server start script:

-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 

This will cause the JVM to listen on the specified port, then from your IDE you can start a remote debug session against that port and debug as if the code was running in the same process.

Working this way prevent you restarting the server so frequently and hence side-steps your problem with Websphere's start-up time.

You can get some odd results if the binaries on the server and the source in the IDE get out of sync but on the whole that's not a problem.

Answer 2

One of the main reasons is that you have a large application with many modules, classes, manifests, XML descriptors so on, and the fact that Websphere application server start up process is single threaded per se (thus each application may be started in a separate thread if they has equal weight). One other reason is that the Eclipse EMF and JST frameworks are very I/O intensive during startup and publish/deploy.

One other reason for the tedious start up is the annotation scanning which will occur during publish/deploy. This annotation scanning can be controlled and modified in a various ways. Look at this site: http://wasdynacache.blogspot.se/2012/05/how-to-speed-up-annotation-processing.html

First of all, examine and evaluate your hardware, both CPU, memory and HDD. Is your processor/s running 100% for a long time during start up? If so, the processor may be too weak. Is paging occur? then you may have to put in some more RAM. The Websphere/eclipse JST and EMF frameworks are very I/O intense so you should consider to invest in a SSD disc. You should also make sure that other processes on your machine (virus protection software etc.) don´t steal hardware resources from the Websphere java processes.

So for the hardware: 1. Processor - a pretty fast one, since the publish and the startup is mostly singlethreaded you do not need that many cpu cores 2. Memory - You will at least need 512Mb of physical RAM, this depends of the size of your application of course. 3. Storage - I would definitely go for a fast SSD since the underlying eclipse framework is I/O intensive.

Here are some tricks to reduce the footprint of the start up phase. Please before applying these settings make sure that you record a baseline start up so that you can observe the difference in start up, i.e. the reduced start up time.

1. JVM args : -Xverify:none -Xquickstart -Xnoclassgc -XX:+UseNUMA -XtlhPrefetch -Xgcthreads4 (I got 4 virtual processors installed on my machine)
2. Extend the heap size to match the demands of your application.
3. Disable the autostart of the application to reduce publish time.
4. Disable PMI and unnecessary tracing.
5. Profile your application during startup and fix bottlenecks if found any.

Other JVM arguments that may gain performance:

• com.ibm.cacheLocalHost=true
• com.ibm.ws.classloader.zipFileCacheSize=512
• com.ibm.ws.classloader.resourceRequestCacheSize=1024  
• com.ibm.ws.management.event.pull_notification_timeout =20000
• com.ibm.ws.amm.scan.context.filter.packages=true
• org.eclipse.jst.j2ee.commonarchivecore.disableZip=true

Jvm arguments that will make the Websphere application server to stop immediately:

• com.ibm.ejs.sm.server.quiesceTimeout=0
• com.ibm.ejs.sm.server.quiesceInactiveRequestTime=1000

Webcontainer properties:

• com.ibm.wsspi.jsp.disableTldSearch=true
• com.ibm.wsspi.jsp.disableResourceInjection=true

JVM arguments that may be specified eclipse.ini (Note that the heap parameters is configured according to the conditions of my environment)

• -Dcom.ibm.ws.management.event.max_polling_interval=5000
• -Xquickstart
• -Xverify:none
• -Xmxcl25000
• -Xjit:dataTotal=65536
• -Xcodecache64m
• -Xscmx48m
• -Xnolinenumbers
• -Xverify:none
• -Xmnx64m
• -Xmx1446m
• -Xmnx64m
• -XX:+UseCompressedOops
• -XX:+UseNUMA

Answer 3

5 to 6 mins is not normal. I use RAD and WAS everyday and get decent startup times. Which version of WAS are you running and how much RAM do you have?

If you share several workspaces and projects for a same WAS profile, consider creating a new WAS profile for your workspace.

You probably tried that but here's a simple check list of things to try on first hand. Make sure that your server settings in RAD has the following options enabled:

• Optimize server for testing and developing
• Run server with resources on the workspace
• Minimize application files copied to the server

Uncheck "Enable universal test client" if you don't need it.

In the admin console you can verify some server settings such as

• Run in development mode
• Parallel start
• Start components as needed

You can also uninstall the ivt app that comes installed by default when creating a new WAS profile. Then the usual things such as a drive that is not too fragmented and a pagefile size that is properly set.

And one last thing that you probably know already, republish to your server instead of restarting it.