How to assess the risk of a java version upgrade?

Question

I'm being asked to assess whether we can safely upgrade the java version on one of our production-deployed webapps. The codebase is fairly large and we want to avoid having to regression test everything (no automated tests sadly), but we've already encountered at least one problem during some manual testing (XmlStringReader.getLocalName now throws an IllegalStateExeption when it just used to return null) and higher-ups are pretty nervous about the upgrade.

The current suggested approach is to do a source compare of the JDK sources for each version and assess those changes to see which ones might have impact, but it seems there's a lot of changes to go through (and as mentioned the codebase is kinda large). Is it safe and easier to just review the java version changes for each version? Or is there an easier way to conduct this assessment?

Edit: I forgot to mention the version upgrade being considered is a minor version upgrade, i.e. 1.6.10 to 1.6.33

Answer 1

Nothing is going to replace testing it in a real system. you may be able to catch something blatant in a bug report or in visual inspection, but detecting changes due to more complex interactions will be impossible. or even detecting seemingly simple changes which alter how the GC affects your running app or how hotspot optimizes your code (you are checking the c++ code too, right) or how some key algorithm performs...