Guess method line number from JVM fatal error log

Question

I have a fatal error log without a core-dump and need to identify the cause. This is the stack found in the .log file:

# Problematic frame:
# C  [libc.so.6+0x7b4bb]  memcpy+0x15b

{...}

Stack: [0x00002ac8c4d2c000,0x00002ac8c4e2d000],  sp=0x00002ac8c4e28ef8,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x7b4bb]  memcpy+0x15b
C  [libzip.so+0x50b0]  ZIP_GetEntry+0xd0
C  [libzip.so+0x3eed]  Java_java_util_zip_ZipFile_getEntry+0xad
J 28  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002ac8acf404ee [0x00002ac8acf40420+0xce]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 28  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002ac8acf40478 [0x00002ac8acf40420+0x58]
J 33 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (86 bytes) @ 0x00002ac8acf45548 [0x00002ac8acf45480+0xc8]
J 58 C2 java.util.jar.JarFile.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry; (9 bytes) @ 0x00002ac8acf4e828 [0x00002ac8acf4e7e0+0x48]
J 44 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (91 bytes) @ 0x00002ac8acf47168 [0x00002ac8acf47100+0x68]
J 34 C2 sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (74 bytes) @ 0x00002ac8acf41f70 [0x00002ac8acf41f00+0x70]
j  java.net.URLClassLoader$1.run()Ljava/lang/Class;+26
j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
j  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
j  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+70
j  sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+36
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
v  ~StubRoutines::call_stub
j  com.smackdapp.SmackHandler.handleRequest(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+72

As I have read in another StackOverflow answers, this can be a common error while replacing any .jar that the jvm needs to load (or a hardware memory error).

What I'm trying to guess is the .jar file that caused this fatal error. Is there any way to identify the line number in my source code with the information in the fatal error log?

I was hoping that this "V+72" at the end of the line has something to do, but can't figure out.

Answer 1

This may not help to solve the problem, eventually, but it answers a part of the question:

As already guessed by Hot Licks in his comment, the +72 just is the bytecode offset. I tested this with a small program:

^{(otherwise unrelated, just had it quickly available here from another question)}

class TestNativeArray3D
{
    public static void main(String args[])
    {
        System.loadLibrary("TestNativeArray3D");
        int terrain[][][] = genTerrain(123, 8, 6);
    }
    private native static int[][][] genTerrain(int seed, int x, int y);
}

The native genTerrain function blatantly creates an arbitrary error, by calling

jclass errorClass = env->FindClass("XXX");
env->NewObjectArray(10, errorClass, NULL);

causing a core dump.

The stack looks as follows:

Stack: [0x0000000002500000,0x0000000002600000],  sp=0x00000000025ff4e0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x14ebb6]
C  [TestNativeArray3D.dll+0x397d]  JNIEnv_::NewObjectArray+0x4d
C  [TestNativeArray3D.dll+0x3ae8]  Java_TestNativeArray3D_genTerrain+0x98
C  0x0000000002715534

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  TestNativeArray3D.genTerrain(III)[[[I+0
j  TestNativeArray3D.main([Ljava/lang/String;)V+11
v  ~StubRoutines::call_stub

Here, the line that analogously contains the (possible) offset is

j TestNativeArray3D.main([Ljava/lang/String;)V+11

When decompiling the class file with

javap -c -l TestNativeArray3D

(note the -l (small "L") to obtain the line numbers!) the output is

class TestNativeArray3D {
  TestNativeArray3D();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return
    LineNumberTable:
      line 3: 0

  public static void main(java.lang.String[]);
    Code:
       0: ldc           #2                  // String TestNativeArray3D
       2: invokestatic  #3                  // Method java/lang/System.loadLibrary:(Ljava/lang/String;)V
       5: bipush        123
       7: bipush        8
       9: bipush        6
      11: invokestatic  #4                  // Method genTerrain:(III)[[[I
      14: astore_1
      15: return
    LineNumberTable:
      line 7: 0
      line 8: 5
      line 9: 15
}

and indeed, the native call happens at offset 11. (And yes, I counter-checked this by adding some further code before this call: The offset in the core dump changes accordingly).

The "LineNumberTable" will allow you to map between bytecode offsets and lines. The table in the example means that

• source code line 7 corresponds to bytecode offset 0,
• source code line 8 corresponds to bytecode offset 5,
• source code line 9 corresponds to bytecode offset 15

so the critical instruction here is related to code line 8. (This was obvious, in this case, because this was directly the invokestatic call. I just wanted to point out that you can look up the LineNumberTable for the mapping of the bytecode offset to the actual line number in the source code)

---

Unfortunately, this will hardly help you to really solve the error, because this one was obviously caused some levels deeper, in the native code of libc.so, during some memcpy - which obviously received invalid pointers from the libzip.so...