Scanning classpath/modulepath in runtime in Java 9

Question

I can't seem to find any info on whether scanning all available classes (for interfaces, annotations etc) is still possible in runtime, the way Spring, Reflections and many other frameworks and libraries currently do, in the face of Jigsaw related changes to the way classes are loaded.

EDIT: This question is about scanning the real physical file paths looking for classes. The other question is about dynamically loading classes and resources. It's related but very much not a duplicate.

UPDATE: Jetty project has made a JEP proposal for a standardized API for this. If you have a way to help make this reality, please do. Otherwise, wait and hope.

UPDATE 2: Found this relevant sounding post. Quoting the code snippet for posterity:

If you are really just looking to get at the contents of the modules in the boot layer (the modules that are resolved at startup) then you'll do something like this:

  ModuleLayer.boot().configuration().modules().stream()          .map(ResolvedModule::reference)          .forEach(mref -> {              System.out.println(mref.descriptor().name());              try (ModuleReader reader = mref.open()) {                  reader.list().forEach(System.out::println);             } catch (IOException ioe) {                  throw new UncheckedIOException(ioe);              }          }); 

Answer 1

The following code achieves module path scanning in Java 9+ (Jigsaw / JPMS). It finds all classes on the callstack, then for each class reference, calls classRef.getModule().getLayer().getConfiguration().modules(), which returns a a List<ResolvedModule>, rather than just a List<Module>. (ResolvedModule gives you access to the module resources, whereas Module does not.) Given a ResolvedModule reference for each module, you can call the .reference() method to get the ModuleReference for a module. ModuleReference#open() gives you a ModuleReader, which allows you to list the resources in a module, using ModuleReader#list(), or to open a resource using Optional<InputStream> ModuleReader#open(resourcePath) or Optional<ByteBuffer> ModuleReader#read(resourcePath). You then close the ModuleReader when you're done with the module. This is not documented anywhere that I have seen. It was very difficult to figure all this out. But here is the code, in the hope that someone else will benefit from this.

Note that even in JDK9+, you can still utilize traditional classpath elements along with module path elements, so for a complete module path + classpath scan, you should probably use a proper classpath scanning solution, such as ClassGraph, which supports module scanning using the below mechanism (disclaimer, I am the author). You can find a reflection-based version of the following code here.

Also note that there was a bug in StackWalker in several JDK releases after JDK 9 that has to be worked around, see the above reflection-based code for details.

package main;  import java.lang.StackWalker; import java.lang.StackWalker.Option; import java.lang.StackWalker.StackFrame; import java.lang.module.ModuleReader; import java.lang.module.ModuleReference; import java.lang.module.ResolvedModule; import java.net.URI; import java.security.AccessController; import java.security.PrivilegedAction; import java.util.AbstractMap.SimpleEntry; import java.util.ArrayDeque; import java.util.ArrayList; import java.util.Collections; import java.util.Deque; import java.util.HashSet; import java.util.List; import java.util.Map.Entry; import java.util.Optional; import java.util.Set; import java.util.stream.Stream;  public class Java9Scanner {      /** Recursively find the topological sort order of ancestral layers. */     private static void findLayerOrder(ModuleLayer layer,             Set<ModuleLayer> visited, Deque<ModuleLayer> layersOut) {         if (visited.add(layer)) {             List<ModuleLayer> parents = layer.parents();             for (int i = 0; i < parents.size(); i++) {                 findLayerOrder(parents.get(i), visited, layersOut);             }             layersOut.push(layer);         }     }      /** Get ModuleReferences from a Class reference. */     private static List<Entry<ModuleReference, ModuleLayer>> findModuleRefs(             Class<?>[] callStack) {         Deque<ModuleLayer> layerOrder = new ArrayDeque<>();         Set<ModuleLayer> visited = new HashSet<>();         for (int i = 0; i < callStack.length; i++) {             ModuleLayer layer = callStack[i].getModule().getLayer();             findLayerOrder(layer, visited, layerOrder);         }         Set<ModuleReference> addedModules = new HashSet<>();         List<Entry<ModuleReference, ModuleLayer>> moduleRefs = new ArrayList<>();         for (ModuleLayer layer : layerOrder) {             Set<ResolvedModule> modulesInLayerSet = layer.configuration()                     .modules();             final List<Entry<ModuleReference, ModuleLayer>> modulesInLayer =                     new ArrayList<>();             for (ResolvedModule module : modulesInLayerSet) {                 modulesInLayer                         .add(new SimpleEntry<>(module.reference(), layer));             }             // Sort modules in layer by name for consistency             Collections.sort(modulesInLayer,                     (e1, e2) -> e1.getKey().descriptor().name()                             .compareTo(e2.getKey().descriptor().name()));             // To be safe, dedup ModuleReferences, in case a module occurs in multiple             // layers and reuses its ModuleReference (no idea if this can happen)             for (Entry<ModuleReference, ModuleLayer> m : modulesInLayer) {                 if (addedModules.add(m.getKey())) {                     moduleRefs.add(m);                 }             }         }         return moduleRefs;     }      /** Get the classes in the call stack. */     private static Class<?>[] getCallStack() {         // Try StackWalker (JDK 9+)         PrivilegedAction<Class<?>[]> stackWalkerAction =                 (PrivilegedAction<Class<?>[]>) () ->                     StackWalker.getInstance(                             Option.RETAIN_CLASS_REFERENCE)                     .walk(s -> s.map(                             StackFrame::getDeclaringClass)                             .toArray(Class[]::new));         try {             // Try with doPrivileged()             return AccessController                     .doPrivileged(stackWalkerAction);         } catch (Exception e) {         }         try {             // Try without doPrivileged()             return stackWalkerAction.run();         } catch (Exception e) {         }          // Try SecurityManager         PrivilegedAction<Class<?>[]> callerResolverAction =                  (PrivilegedAction<Class<?>[]>) () ->                     new SecurityManager() {                         @Override                         public Class<?>[] getClassContext() {                             return super.getClassContext();                         }                     }.getClassContext();         try {             // Try with doPrivileged()             return AccessController                     .doPrivileged(callerResolverAction);         } catch (Exception e) {         }         try {             // Try without doPrivileged()             return callerResolverAction.run();         } catch (Exception e) {         }          // As a fallback, use getStackTrace() to try to get the call stack         try {             throw new Exception();         } catch (final Exception e) {             final List<Class<?>> classes = new ArrayList<>();             for (final StackTraceElement elt : e.getStackTrace()) {                 try {                     classes.add(Class.forName(elt.getClassName()));                 } catch (final Throwable e2) {                     // Ignore                 }             }             if (classes.size() > 0) {                 return classes.toArray(new Class<?>[0]);             } else {                 // Last-ditch effort -- include just this class                 return new Class<?>[] { Java9Scanner.class };             }         }     }      /**      * Return true if the given module name is a system module.      * There can be system modules in layers above the boot layer.      */     private static boolean isSystemModule(             final ModuleReference moduleReference) {         String name = moduleReference.descriptor().name();         if (name == null) {             return false;         }         return name.startsWith("java.") || name.startsWith("jdk.")             || name.startsWith("javafx.") || name.startsWith("oracle.");     }      public static void main(String[] args) throws Exception {         // Get ModuleReferences for modules of all classes in call stack,         List<Entry<ModuleReference, ModuleLayer>> systemModuleRefs = new ArrayList<>();         List<Entry<ModuleReference, ModuleLayer>> nonSystemModuleRefs = new ArrayList<>();          Class<?>[] callStack = getCallStack();         List<Entry<ModuleReference, ModuleLayer>> moduleRefs = findModuleRefs(                 callStack);         // Split module refs into system and non-system modules based on module name         for (Entry<ModuleReference, ModuleLayer> m : moduleRefs) {             (isSystemModule(m.getKey()) ? systemModuleRefs                     : nonSystemModuleRefs).add(m);         }          // List system modules         System.out.println("\nSYSTEM MODULES:\n");         for (Entry<ModuleReference, ModuleLayer> e : systemModuleRefs) {             ModuleReference ref = e.getKey();             System.out.println("  " + ref.descriptor().name());         }          // Show info for non-system modules         System.out.println("\nNON-SYSTEM MODULES:");         for (Entry<ModuleReference, ModuleLayer> e : nonSystemModuleRefs) {             ModuleReference ref = e.getKey();             ModuleLayer layer = e.getValue();             System.out.println("\n  " + ref.descriptor().name());             System.out.println(                     "    Version: " + ref.descriptor().toNameAndVersion());             System.out.println(                     "    Packages: " + ref.descriptor().packages());             System.out.println("    ClassLoader: "                     + layer.findLoader(ref.descriptor().name()));             Optional<URI> location = ref.location();             if (location.isPresent()) {                 System.out.println("    Location: " + location.get());             }             try (ModuleReader moduleReader = ref.open()) {                 Stream<String> stream = moduleReader.list();                 stream.forEach(s -> System.out.println("      File: " + s));             }         }     } } 

Answer 2

The actual issue here is to find the paths to all jars and folders on the classpath. Once when you have them, you can scan.

What I did is the following:

• get the current module descriptor for current class
• get all requires modules
• for each such module open resource of MANIFEST.MF
• remove the MANIFEST.MF path from the resource url
• what remains is the classpath of the module, i.e. to it's jar or folder.

I do the same for current module, to get the classpath for current code.

This way I collect classpath of a currently working module and all its required modules (1 step away). That was working for me - and my Java8 scanner was still being able to do the job. This approach does not require any additional VM flag etc.

I could extend this approach to get all required modules easily (not only the first level), but for now, I don't need that.

Code.