How to make JavaCompiler.CompilationTask use custom ClassLoader or use .class files for missin .java files?

Question

More spicific: I havewritten my own ClassLoader, that loads

1. .jar files from global library path
2. .jar files from project specific path
3. compiles .java files in project specific path
4. loads all .class files in project specific path

So far, this separates my project instances, loads everything fine from all (sub)directories, works for (1) all libraries and (2) sub-libraries, (3) can compile all .java files, (4) can load .class files, and I also can re-instanciate classes that are already loaded, because my ClassLoader manages sub-ClassLoaders to allow this.

Now, what I want to improve is that at (3) when I call the compiler, I do not wanna re-compile every .java file in the directory, but only those, where the corresponding .class file does not exist or has the wrong timestamp.

So I only pass those .java files, that need re-compiling, not all, with the consequence, that the compiler cannot find all needed classes (inside those .java files that I didnt pass him to compile). Rather, the compiler shall get its missing compilation information (.class files instad of .java files) from my ClassLoader, that has already loaded those .class files.

To realize that, I have implemented my own FileManager, that I pass to JavaCompiler.getTask(). Within that custom FileManager, I return my ClassLoader in FileManager.getClassLoader().

Should look like this:

1. .jar files from global library path
2. .jar files from project specific path
3. compiles only some .java files in project specific path, loading missing class definitions from .class files (loaded by my specific ClassLoader)
4. loads all .class files in project specific path

But when the JavaCompiler.CompilationTask runs, it never accesses my ClassLoader's .loadClass() or .findClass() methods, thus does not find the necessary .class files, thus 'throws' me a compilation error. (i.e. I get the diagnostics, and transform them into an Exception)

So, my actual question is:

• Is my concept broken?
• Is it possible at all? Or is the compiler unable to use .class files instead of .java files?
• Is there any mean trick to it?

---

OK, so I'm gonna post parts of the code I wrote, but this uses lots of my libraries and lots more internal classes, so you will not get this code to work!

Also note, this is much more convoluted, because I store lots of additional information, that would not be needed for the task I asked for.

So use this as a guideline. Basically, you can leave out all the information handling where I store data in Maps/Multimaps, they're mostly for tracing resources.

Also, JcDirLoader does not have to extend ClassLoader for the job, that is also designed for a "higher" purpose ;-) And the way that the ClassLoaders call each other can also be drastically simplified!

package jc.lib.lang.reflect.loader;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashSet;

import com.sun.org.apache.xalan.internal.xsltc.compiler.CompilerException;

import jc.lib.collection.list.JcList;
import jc.lib.collection.map.JcHashMap;
import jc.lib.collection.map.JcMultiMap;
import jc.lib.io.files.finder.JcFileFinder;
import jc.lib.lang.JcUFile;
import jc.lib.lang.JcUFileType;
import jc.lib.lang.reflect.classfileanalyzer.JcClassFileInfo;
import jc.lib.lang.reflect.compiler.JcJavaFileCompiler;
import jc.lib.lang.reflect.loader.classes.JcClassFileLoader;
import jc.lib.lang.reflect.loader.classes.JcUClassLoader;
import jc.lib.lang.reflect.loader.jars.JcUJarfileLoader;
import jc.lib.lang.reflect.loader.util.ClassName;
import jc.lib.lang.reflect.loader.util.ClassState;
import jc.lib.lang.reflect.loader.util.JcClassLoaderInfo;



public class JcDirLoader extends ClassLoader {



    private final JcHashMap<File, ClassName>                mFile2Classname         = new JcHashMap<>();
    private final JcHashMap<ClassName, JcClassLoaderInfo>   mClassname2Classinfo    = new JcHashMap<>();

    private final JcMultiMap<ClassName, JcClassFileLoader>  mClassname2Loaders      = new JcMultiMap<>();
    private final ArrayList<JcClassFileLoader>              mAvailableFileLoaders   = new ArrayList<>();

    private final ClassLoader mParentLoader;

    public JcDirLoader() {
        //      super(JcUClassLoader.getThreadContextClassLoader());
        // JcDirLoader.class.getClassLoader();
        mParentLoader = JcUClassLoader.getThreadContextClassLoader();
    }



    public JcList<JcClassLoaderInfo> getLoadedClasses() {
        final JcList<JcClassLoaderInfo> ret = new JcList<>(mClassname2Classinfo.values());
        return ret;
    }

    public Class<?> forName(final ClassName pClassName) throws ClassNotFoundException {
        final JcClassLoaderInfo ret = mClassname2Classinfo.get(pClassName);
        if (ret != null) return ret.mClass;

        final Class<?> ret2 = mParentLoader.loadClass(pClassName.toString());
        return ret2;
    }

    public JcClassLoaderInfo getClassInfo(final ClassName pClassName) {
        return mClassname2Classinfo.get(pClassName);
    }
    public ArrayList<JcClassLoaderInfo> getClassInfos() {
        return new ArrayList<>(mClassname2Classinfo.values());
    }



    public void loadDirectory(final File pDir) throws ClassNotFoundException, IOException, CompilerException {
        // collect all files
        final JcList<File> files = JcFileFinder.findInDir(pDir, true);
        final JcList<File> jarFiles = new JcList<>();
        final JcList<File> classFiles = new JcList<>();
        final JcList<File> javaFiles = new JcList<>();
        for (final File file : files) {
            if (JcUFileType.isJavaPackedFile(file)) jarFiles.addItem(file);
            if (JcUFileType.isJavaClassFile(file)) classFiles.addItem(file);
            if (JcUFileType.isJavaSourceFile(file)) javaFiles.addItem(file);
        }

        // handle .jar files
        handleJars(jarFiles);

        // compile .java files
        final boolean reloadNecessary = handleJavaFiles(javaFiles, jarFiles);
        if (reloadNecessary) {
            classFiles.removeAllItems();
            final JcList<File> files2 = JcFileFinder.findInDir(pDir, true);
            for (final File file : files2) {
                if (JcUFileType.isJavaClassFile(file)) classFiles.addItem(file);
            }
        }

        // handle .class files
        handleClassFiles(classFiles);
    }



    /*
     * .jar files
     */
    private void handleJars(final JcList<File> pAvailableJarFiles) throws MalformedURLException, IOException {
        System.out.println("\tLoading " + pAvailableJarFiles.getItemCount() + " .jar files:");
        final JcList<JcClassLoaderInfo> loadedClasses = JcUJarfileLoader.loadJars(pAvailableJarFiles);
        for (final JcClassLoaderInfo ci : loadedClasses) {
            mFile2Classname.put(ci.mContainingFile, ci.mClassName);
            mClassname2Classinfo.put(ci.mClassName, ci);
        }
        for (final File file : pAvailableJarFiles) {
            System.out.println("\t\t" + file + " OK");
        }
        System.out.println("\t\tAll OK");
    }



    /*
     * .class files
     */
    private ArrayList<Class<?>> handleClassFiles(final JcList<File> pClassFiles) throws FileNotFoundException, IOException, ClassNotFoundException {
        System.out.println("\tLoading " + pClassFiles.getItemCount() + " .class files:");

        for (final File file : pClassFiles) {
            final JcClassFileInfo cfi = new JcClassFileInfo(file);
            final ClassName className = ClassName.fromClassFileInfo(cfi);

            final JcClassLoaderInfo info = new JcClassLoaderInfo(null, file, className, null, ClassState.CLASS_FILE, file.lastModified(), null);
            mFile2Classname.put(file, className);
            mClassname2Classinfo.put(className, info);
        }

        final ArrayList<Class<?>> ret = new ArrayList<>();
        for (final File file : pClassFiles) {
            final Class<?> cls = handleClassFile(file);
            ret.add(cls);
        }

        System.out.println("\t\t" + pClassFiles.getItemCount() + " .class files loaded.");
        return ret;
    }
    private Class<?> handleClassFile(final File pFile) throws FileNotFoundException, IOException, ClassNotFoundException {
        final JcClassFileInfo cfi = new JcClassFileInfo(pFile);
        final ClassName className = ClassName.fromClassFileInfo(cfi);
        final JcClassLoaderInfo existing = mClassname2Classinfo.get(className);
        if (!needsReloading(existing, pFile)) return existing.mClass;

        final JcClassFileLoader fileLoader = getCompatibleFileLoader(className);
        fileLoader.setInfo(pFile, className);
        final Class<?> c = fileLoader.loadClass(className.toString());

        final JcClassLoaderInfo info = new JcClassLoaderInfo(null, pFile, className, c, ClassState.CLASS_FILE, pFile.lastModified(), fileLoader);
        mFile2Classname.put(pFile, className);
        mClassname2Classinfo.put(className, info);

        return c;
    }
    static private boolean needsReloading(final JcClassLoaderInfo pExisting, final File pNewFile) {
        if (pExisting == null) return true;
        if (pExisting.mClass == null) return true;
        if (!pNewFile.equals(pExisting.mContainingFile)) return true;
        if (pNewFile.lastModified() != pExisting.mContainingFile.lastModified()) return true;
        return false;
    }



    /*
     * .java files
     */
    private boolean handleJavaFiles(final JcList<File> pJavaFiles, final JcList<File> pAvailableJarFiles) throws IOException, CompilerException {
        System.out.println("\tChecking " + pJavaFiles.getItemCount() + " .java files:");

        final boolean recompile = needRecompiling(pJavaFiles);
        if (!recompile) {
            System.out.println("\t\tNo Java files needed recompiling.");
            return false;
        }

        final JcList<File> javaFilesToCompile = pJavaFiles;

        // info
        System.out.print("\t\tRecompiling files: ");
        for (final File file : javaFilesToCompile) {
            System.out.print(file.getName() + " ");
        }
        System.out.println();

        // recompile
        final String[] jarFileNames = new String[pAvailableJarFiles.getItemCount()];
        for (int i = 0; i < pAvailableJarFiles.getItemCount(); i++) {
            final File jarFile = pAvailableJarFiles.getItem(i);
            jarFileNames[i] = jarFile.getAbsolutePath();
        }
        JcJavaFileCompiler.compileFiles(pJavaFiles.toArray(), this, jarFileNames);

        // set time of compiled files to match dates
        for (final File f : pJavaFiles) {
            final long timestamp = f.lastModified();
            final File cls = getClassfileForSourcefile(f);
            cls.setLastModified(timestamp);
        }

        // return results
        final JcList<JcClassLoaderInfo> ret = new JcList<>();
        for (final File file : javaFilesToCompile) {
            final File classFile = getClassfileForSourcefile(file);
            if (!classFile.exists()) throw new FileNotFoundException("File '" + classFile.getAbsolutePath() + "' could not be found, but was compiled from '" + file.getAbsolutePath() + "'!");

            final JcClassLoaderInfo ci = new JcClassLoaderInfo(file, classFile, null, null, ClassState.JAVA_FILE, classFile.lastModified(), null);
            ret.addItem(ci);
        }

        System.out.println("\t\t" + pJavaFiles.getItemCount() + " Java files recompiled.");
        return true;
    }
    static private boolean needRecompiling(final JcList<File> pJavaFiles) {
        for (final File file : pJavaFiles) {
            if (!JcUFileType.isJavaSourceFile(file)) continue;

            final long sourceDate = file.lastModified();
            final File classFile = getClassfileForSourcefile(file);
            final long classDate = !classFile.exists() ? 0 : classFile.lastModified();
            if (sourceDate > classDate) return true;
        }

        return false;
    }

    private JcClassFileLoader getCompatibleFileLoader(final ClassName pClassName) {
        // check if can re-use another existing loader
        final HashSet<JcClassFileLoader> oldLoaders = mClassname2Loaders.getUniqueValues(pClassName);
        for (final JcClassFileLoader loader : mAvailableFileLoaders) {
            if (oldLoaders.contains(loader)) continue;

            mClassname2Loaders.put(pClassName, loader);
            //          System.out.println("\tUsing " + loader + " for " + pClassName);
            return loader;
        }

        // create new loader
        final JcClassFileLoader newLoader = new JcClassFileLoader(mParentLoader, this);
        mAvailableFileLoaders.add(newLoader);
        mClassname2Loaders.put(pClassName, newLoader);
        //      System.out.println("Created new " + newLoader + " for " + pClassName);
        return newLoader;
    }



    static public File getClassfileForSourcefile(final File pSourceFile) {
        final String classFilename = JcUFile.toString(pSourceFile, true, true, true, false) + JcUFileType.CLASS_EXTENSION;
        final File classFile = new File(classFilename);
        return classFile;
    }



    @Override public Class<?> loadClass(final String pClassname) throws ClassNotFoundException {
        System.out.println(" -> JcDirLoader.loadClass(" + pClassname + ")");
        try {
            final ClassName cn = ClassName.fromString(pClassname);
            final JcClassLoaderInfo ci = getClassInfo(cn);
            if (ci == null) return mParentLoader.loadClass(pClassname);

            final File f = ci.mContainingFile;
            final Class<?> cls = handleClassFile(f);
            return cls;

        } catch (final Exception e) {
            throw new ClassNotFoundException(pClassname, e);
        }
    }

    @Override public Class<?> findClass(final String pClassname) throws ClassNotFoundException {
        System.out.println(" -> JcDirLoader.findClass(" + pClassname + ")");

        final ClassName cn = ClassName.fromString(pClassname);
        final JcClassLoaderInfo ci = getClassInfo(cn);
        if (ci != null) return ci.mClassLoader.loadClass(pClassname);

        final Class<?> test = loadClass(pClassname);
        if (test != null) return test;

        System.out.println("XXX -> " + pClassname);
        return super.findClass(pClassname);
    }

    @Override public URL getResource(final String pName) {
        return super.getResource(pName);
    }



}

and more code

public class JcUJarfileLoader {

    static public JcList<JcClassLoaderInfo> loadJars(final JcList<File> pJarFiles) throws MalformedURLException, IOException {
        final JcList<JcClassLoaderInfo> ret = new JcList<>();
        if (pJarFiles == null || pJarFiles.getItemCount() < 1) return ret;

        // convert files to URL to make all jars available to all requests
        final ArrayList<URL> urls = new ArrayList<>(pJarFiles.getItemCount());
        for (final File jarFile : pJarFiles) {
            final URL url = jarFile.toURI().toURL();
            urls.add(url);
        }
        final URL[] urlArr = urls.toArray(new URL[0]);

        // iterate through jar, load all inner classes
        try (final URLClassLoader classLoader = new URLClassLoader(urlArr);) { //
            for (final File jarFile : pJarFiles) {
                try (final JarFile file = new JarFile(jarFile);) {
                    final Enumeration<JarEntry> entries = file.entries();
                    while (entries.hasMoreElements()) {
                        final JarEntry jarEntry = entries.nextElement();
                        if (!JcUFileType.isJavaClassFile(jarEntry)) continue;

                        try {
                            //                          System.out.println("reloading1 " + jarEntry);
                            final ClassName className = ClassName.fromZipEntry(jarEntry);
                            final Class<?> cls = classLoader.loadClass(className.toString());
                            final JcClassLoaderInfo i = new JcClassLoaderInfo(null, jarFile, className, cls, ClassState.CLASS_FILE_IN_JAR, jarFile.lastModified(), classLoader);
                            ret.addItem(i);

                        } catch (final ClassNotFoundException e2) {
                            System.out.println("JcUJarfileLoader.reloadJar(e2) " + e2);
                        }
                    }
                }
            }
        }

        return ret;
    }



}

and more

package jc.lib.lang.reflect.compiler;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.Arrays;

import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;
import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;

import com.sun.org.apache.xalan.internal.xsltc.compiler.CompilerException;

import jc.lib.lang.JcUFile;
import jc.lib.lang.reflect.loader.JcDirLoader;
import jc.lib.lang.string.JcUString;



public class JcJavaFileCompiler {



    static public void compileCode(final String pCode) throws IOException, CompilerException {
        final File tempFile = File.createTempFile("jccompiler_", ".java");
        try {
            JcUFile.writeString_UTF8(tempFile, pCode);
            compileFiles(new File[] { tempFile }, null);

        } finally {
            tempFile.deleteOnExit();
        }
    }

    static public void compileFiles(final File pFiles[], @SuppressWarnings("unused") final JcDirLoader pClassLoader_Nullable, final String... pBindingLibraries) throws IOException, CompilerException {
        if (pFiles == null || pFiles.length < 1) return;
        for (final File f : pFiles) {
            if (!f.exists()) throw new FileNotFoundException(f.getAbsolutePath());
        }

        final DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<>();
        final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
        if (compiler == null) throw new NoClassDefFoundError("ToolProvider.getSystemJavaCompiler() cannot find a compiler! Make sure you're running on JDK or have linked tools.jar into the classpath!");

        try (final StandardJavaFileManager fileManager2 = compiler.getStandardFileManager(diagnostics, null, null);
        /*JcJavaFileManager fileManager = new JcJavaFileManager(pClassLoader_Nullable, fileManager2);*/) {
            String addLibs = "";
            if (pBindingLibraries != null) for (final String l : pBindingLibraries) {
                if (l == null || l.length() < 1) continue;
                addLibs += ";" + l;
            }
            final ArrayList<String> optionList = new ArrayList<>();
            optionList.add("-classpath");
            optionList.add(System.getProperty("java.class.path") + addLibs);

            final Iterable<? extends JavaFileObject> compilationUnit = /*fileManager*/fileManager2.getJavaFileObjectsFromFiles(Arrays.asList(pFiles));
            final JavaCompiler.CompilationTask task = compiler.getTask(null, /*fileManager*/ fileManager2, diagnostics, optionList, null, compilationUnit);
            final boolean done = task.call().booleanValue();
            if (done) return;

            // collect error data and throw
            final StringBuilder sb = new StringBuilder();
            for (final Diagnostic<? extends JavaFileObject> diagnostic : diagnostics.getDiagnostics()) {
                sb.append(diagnostic + "\n");
            }
            throw new CompilerException(sb.toString());

        } catch (final IOException e) {
            e.printStackTrace();
            throw e;
        }
    }

    static public String getClassNameFromJavaCode(final File pJavaFile) throws IOException {
        String code = JcUFile.loadString(pJavaFile);
        code = JcUString.removeJavaComments(code);

        final String pkg = JcUString.getBetween(code, "package ", ";");
        final String name = JcUFile.toString(pJavaFile, false, true, false, false);
        return pkg + "." + name;
    }

    static public Class<?> getClass(final String pFullClassName) throws ClassNotFoundException, IOException {
        try (final URLClassLoader classLoader = new URLClassLoader(new URL[] { new File("./").toURI().toURL() });) {
            final Class<?> loadedClass = classLoader.loadClass(pFullClassName);
            return loadedClass;
        }
    }



}

last file for today

package jc.lib.lang.reflect.loader.classes;

import java.io.File;
import java.io.IOException;

import jc.lib.lang.JcUFile;
import jc.lib.lang.reflect.loader.JcDirLoader;
import jc.lib.lang.reflect.loader.util.ClassName;
import jc.lib.lang.string.JcUString;



/**
 * Our Custom Class Loader to load the classes. Any class in the com.journaldev
 * package will be loaded using this ClassLoader. For other classes, it will
 * delegate the request to its Parent ClassLoader.
 *
 */
public class JcClassFileLoader extends ClassLoader {



    private final JcDirLoader mJcDirLoader;

    private File        mFile;
    private ClassName   mClassname;

    public JcClassFileLoader(final ClassLoader parent, final JcDirLoader pJcDirLoader) {
        super(parent);
        mJcDirLoader = pJcDirLoader;
    }



    public void setInfo(final File pFile, final ClassName pClassname) {
        mFile = pFile;
        mClassname = pClassname;
    }



    @Override public Class<?> loadClass(final String pClassname) throws ClassNotFoundException {
        //      System.out.println(" *** JcClassFileLoader.loadClass(" + pClassname + ") primed with (" + mFile + "," + mClassname + ")");
        if (!JcUString.equals(pClassname, mClassname.toString())) return super.loadClass(pClassname);

        try {
            final byte[] b = JcUFile.readBytes(mFile);
            final Class<?> c = defineClass(mClassname.toString(), b, 0, b.length);
            resolveClass(c);
            //          System.out.println("LOADED: " + c.getSimpleName() + "\t" + c.getName() + "\t" + c.getPackage());
            return c;

        } catch (final LinkageError e) {
            throw new LinkageError("Error while loading file '" + mFile + "' as Class '" + mClassname + "'", e);
        } catch (final IOException e) {
            e.printStackTrace();
            return null;
        }
    }



    @Override protected Class<?> findClass(final String pName) throws ClassNotFoundException {
        System.out.println(" *** JcClassFileLoader.findClass(" + pName + ")");
        return mJcDirLoader.findClass(pName);
    }



}

Answer 1

Eclipse has incremental compiler that can be used standalone, it should be possible to adapt it to your needs. I don't think it is possible to do what you try with standard javac, but would be interesting to know thoughts on that from oracle compiler people.

There is also incremental compiler in Gradle, not sure how easy it will be to reuse it though.