Grails app performance degrades over time

Question

I have noticed that after my Grails app has been deployed for about 2 weeks, performance degrades significantly, and I have to redeploy. I am using the Spring Security plugin and caching users. My first inclination is that it has something to do with this and the session cache size, but I'm not sure how to go about verifying this.

Does it sound like I'm on the right track? Has anyone else experienced this and narrowed down the problem? Any help would be great.

Thanks!

Answer 1

Never guess where to optimize, it's going to be wrong.

Get a heap dump and a profile it a little (VisualVM worked fine for me).

It might be a memory leak, like it happened to me. What is your environment - OS, webserver, Grails?

Answer 2

I would recommend getting YourKit (VisualVM has limited information) and use this to profile your application in production (if possible).

Alternatively you could create a performance test (with JMeter for example) and performance test the pieces of your application that you suspect is causing the performance degredation.

Monitoring memory,cpu,threads,gc and such while running some simple JMeter performance tests will definitely find the culprit. This way you can easily re-test your system over time and see if you have incorporated new "performance killing" bugs.

Performance testing tools/services:

• JMeter
• Grinder
• Selenium (Can performance test with selenium grid, need hw though)
• Browsermob (Commercial, which uses Selenium + Selenium-Grid)
• NeoLoad by NeoTys (Commercial, trial version available)
• HP Loadrunner (Commercial, The big fish on the market, trial version available)

Answer 3

I'd also look into installing the app-info plugin and turning on a bunch of the options (especially around hibernate) to see if things are getting out of control there. Could be something that's filling the hibernate session but never closing a transaction.

Another area to look at is if you're doing anything with the groovy template engine. That has a known memory leak, that's sort of unfixable if you're not caching the class/results. Recently fixed a problem around this on our app. If you're seeing any perm gen errors, this could be the case.