Understanding Eclipse debug source lookup with dynamically compiled and loaded code

Question

Background info:

I have this java framework that is meant to run external scripts. To do this, I use a combination of a classloader and the system java compiler to compile .java "script" files that DO NOT exist on my project's build path. All of this works, compiler black magic and all.

The inherent complication with externally loaded code is the difficulty to debug. I have addressed this by using the remote debugging feature of the java runtime.

So, I have a debug configuration that attaches to my executable jar, which has the directory with the external java scripts on the source lookup path. This actually WORKED for a while.Actually, it never worked properly, I just had scripts accidentally on my build path. Confusingly enough I can put breakpoints in the scripts, and the debugger actually STOPS there (consistent line number, -verbose:class logging and all). Understanding how eclipse finds the source files is something that would help, though. The majority of eclipse documentation is comprised of user manuals, after all.

WHAT I SUSPECTED was that I had accidentally duplicated certain script files, and thus confused the source lookup with an out-of-sync source file. This is not the case, I have since removed the duplicated files and eclipse still is unable to find the source.

What I've tried

• double, triple, quadruple checked the source lookup paths ensuring it includes every relevant directory
• enabled/disabled search subfolders
• enabled/disabled search duplicates
• using an absolute path to the directory instead of a relative workspace path

Workaround

The only workaround here is to add the script files onto the build path of the project, which is unacceptable for me.

What I'm doing now

I am slowly crawling my way through the eclipse open source project base repository looking for the answer. Eclipse, as it turns out, is a pretty big project.

Question

Can anyone provide an accurate algorithmic representation of how the Eclipse source lookup works?

Knowing this, I could possibly figure out a way to force the Eclipse debugger to use the correct path using reflection. As far as I know, there is no technical limitation that prevents dynamically compiled code from being debugged. I know this because my breakpoints are suspending my threads as I expect them to, the source code just doesn't seem to want to load :(

Related research: It seems that this might be linked with how the class is defined with a null CodeSource location, but apparently the proper thing to do when dynamically compiling code into memory is to give the null arg... the question still stands how/why this matters to eclipse's debugger.

Update 4/22 3:30: So I pursued the CodeSource solution linked above. Now, I am seeing that my class IS being loaded from the proper file path location with the -verbose:class switch, but the source lookup is still failing. Breakpoints are still properly caught, but I am greeted with the familiar Source not found red lettering.

Updated 5/6 3:15: I pursued the javap solution discussed in Andrew's answer. Turns out, the source file attribute in my .class bytecode exactly matches a file that WOULD exist on my source lookup path. This confuses me, because this hints towards folder hierarchy having an influence on the source lookup. However, I have created "phantom" package hierarchies representing the "true" packages(as defined at the top of my .java files) and moving my source files to those folders, but the source lookup is still failing when I add those paths to my source lookup path. Any additional insight as to what additional factors play into the source lookup would be huge.

Answer 1

I have a bit of experience in this area, having done some work on debugging dynamically compiled groovy scripts though the JDT. I never got this to work perfectly, and I think it is mostly a limitation of the JDT, which was never designed to handle dynamically compiled code.

TLDR: My guess is that your dynamically compiled scripts have an incorrect source file attribute in the byte code. This attribute is set in the class file by the compiler. See https://en.wikipedia.org/wiki/Java_class_file

Your confusion, I think, is that the debugger correctly stops at breakpoints you set in the scripts, but the IDE cannot load the source. This is confusing of course, but there is a good explanation for this.

Breakpoints are actually handled by the VM and the VM keeps track of them through a fully qualified name and a line number. This allows breakpoints to be hit regardless of which classloader loads the class file, but it can lead to some confusion if multiple class files are loaded through different classloaders with the same qualified name, but different source code. This algorithm for determining when to stop the VM has nothing to do with actually looking for source code when the VM stops.

Looking for source code is handled by the IDE. Since even in the statically compiled world, the source file name may not match with the class name (inner classes, anonymous classes, etc). The class name cannot be used to look up the source file.

Here is a simplification of what the IDE does when it stops at a breakpoint:

1. Find the class file at that breakpoint
2. Get the source attribute
3. Find a source file in the source lookup path that matches the name of the source attribute
4. Use some heuristics if multiple source files of same name are found (I think this would be the ranking in the source lookup tab)
5. Return the most appropriate source file.

(Caveat, I think that the source attribute is only the simple name of the source file (ie- no directory), so I think the IDE converts the package name to a directory structure as part of the lookup, but I might be wrong about that).

So, the lookup will fail if your dynamically compiled script does not have a proper source attribute. You can check this theory by looking at the byte code. You will have to somehow compile a script and save the bits to disk. Then you can run javap -v myScript on it. I would bet that this is the problem. I have seen this happen before in other dynamically compiled languages.