What is the proper way to instrument classes loaded by bootstrap / extension class loader?

Question

I finally wrote a Java agent with Byte Buddy which uses the Advice API to print a message upon entering and leaving a method. With my current configuration, this agent appears to apply on classes loaded by the Application ClassLoader only.

However, I would like it to apply also to classes loaded by any classloader. I have come across multiple techniques (see enableBootstrapInjection() or ignore()) which does not seem to work. Indeed, enableBootstrapInjection() has disappeared from ByteBuddy, and the ignore() method makes my JVM panic, as I believe I have circular issues like trying to instrument the java.lang.instrument class (but this does not seem to be the only issue, and I cannot find a way to list those errors).

Here is a simplified version of my agent:

AgentBuilder mybuilder = new AgentBuilder.Default()
        .ignore(nameStartsWith("net.bytebuddy."))
        .disableClassFormatChanges()
        .with(RedefinitionStrategy.RETRANSFORMATION)
        .with(InitializationStrategy.NoOp.INSTANCE)
        .with(TypeStrategy.Default.REDEFINE);
mybuilder.type(nameMatches(".*").and(not(nameMatches("^src.Agent")))) // to prevent instrumenting itself
        .transform((builder, type, classLoader, module) -> {
            try {
                return builder
                .visit(Advice.to(TraceAdvice.class).on(isMethod()));
                } catch (SecurityException e) {
                    e.printStackTrace();
                    return null;
                }
            }
        ).installOn(inst);
System.out.println("Done");

and a simplified version of my Advice class, if necessary :

public class TraceAdvice {
    @Advice.OnMethodEnter
    static void onEnter(
        @Origin Method method,
        @AllArguments(typing = DYNAMIC) Object[] args
    ) {
        System.out.println("[+]");
    }

    @Advice.OnMethodExit
    static void onExit() {
        System.out.println("[-]");
    }
}

I am conscious about the circular dependency of instrumenting java.io.PrintStream.println for instance, I could just ignore such methods (with .and(not(nameMatches("^java.io.PrintStream"))) on line 7 for example).

Answer 1

Here is how you can activate logging and get helpful log output. I am also showing you how to manually retransform an already loaded bootstrap class. Bootstrap classes which are loaded after installing the transformer, will automatically be transformed, as you can also see in the log below.

import net.bytebuddy.agent.ByteBuddyAgent;
import net.bytebuddy.agent.builder.AgentBuilder;
import net.bytebuddy.asm.Advice;

import java.lang.instrument.Instrumentation;
import java.lang.instrument.UnmodifiableClassException;
import java.lang.reflect.Method;
import java.util.Properties;

import static net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy.RETRANSFORMATION;
import static net.bytebuddy.matcher.ElementMatchers.*;

class ByteBuddyInstrumentBootstrapClasses {
  public static void main(String[] args) throws UnmodifiableClassException {
    Instrumentation instrumentation = ByteBuddyAgent.install();
    installTransformer(instrumentation);

    // Use already loaded bootstrap class 'Properties'
    System.out.println("Java version: " + System.getProperties().getProperty("java.version"));
    // Retransform already loaded bootstrap class 'Properties'
    instrumentation.retransformClasses(Properties.class);
    // Use retransformed bootstrap class 'Properties' (should yield advice output)
    System.out.println("Java version: " + System.getProperties().getProperty("java.version"));
  }

  private static void installTransformer(Instrumentation instrumentation) {
    new AgentBuilder.Default()
      .disableClassFormatChanges()
      .with(RETRANSFORMATION)
      // Make sure we see helpful logs
      .with(AgentBuilder.RedefinitionStrategy.Listener.StreamWriting.toSystemError())
      .with(AgentBuilder.Listener.StreamWriting.toSystemError().withTransformationsOnly())
      .with(AgentBuilder.InstallationListener.StreamWriting.toSystemError())
      .ignore(none())
      // Ignore Byte Buddy and JDK classes we are not interested in
      .ignore(
        nameStartsWith("net.bytebuddy.")
          .or(nameStartsWith("jdk.internal.reflect."))
          .or(nameStartsWith("java.lang.invoke."))
          .or(nameStartsWith("com.sun.proxy."))
      )
      .disableClassFormatChanges()
      .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
      .with(AgentBuilder.InitializationStrategy.NoOp.INSTANCE)
      .with(AgentBuilder.TypeStrategy.Default.REDEFINE)
      .type(any())
      .transform((builder, type, classLoader, module) -> builder
        .visit(Advice.to(TraceAdvice.class).on(isMethod()))
      ).installOn(instrumentation);
  }

  public static class TraceAdvice {
    @Advice.OnMethodEnter
    static void onEnter(@Advice.Origin Method method) {
      // Avoid '+' string concatenation because of https://github.com/raphw/byte-buddy/issues/740
      System.out.println("[+] ".concat(method.toString()));
    }

    @Advice.OnMethodExit
    static void onExit(@Advice.Origin Method method) {
      // Avoid '+' string concatenation because of https://github.com/raphw/byte-buddy/issues/740
      System.out.println("[-] ".concat(method.toString()));
    }
  }
}

Console log:

[Byte Buddy] BEFORE_INSTALL net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport@2a54a73f on sun.instrument.InstrumentationImpl@16a0ee18
[Byte Buddy] TRANSFORM com.sun.tools.attach.VirtualMachine [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, module jdk.attach, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM ByteBuddyInstrumentBootstrapClasses [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM com.intellij.rt.execution.application.AppMainV2$1 [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM com.intellij.rt.execution.application.AppMainV2 [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM com.intellij.rt.execution.application.AppMainV2$Agent [jdk.internal.loader.ClassLoaders$AppClassLoader@2626b418, unnamed module @4e07b95f, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.text.resources.cldr.ext.FormatData_de [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.util.resources.provider.LocaleDataProvider [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.util.resources.provider.NonBaseLocaleDataMetaInfo [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM sun.util.resources.cldr.provider.CLDRLocaleDataMetaInfo [jdk.internal.loader.ClassLoaders$PlatformClassLoader@7203c7ff, module jdk.localedata, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formattable [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$Conversion [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$Flags [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$FormatSpecifier [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$FixedString [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.Formatter$FormatString [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.ASCII [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.IntHashSet [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.Matcher [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.regex.MatchResult [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.lang.CharacterData00 [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.lang.StringUTF16$CharsSpliterator [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.stream.IntPipeline$9$1 [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] TRANSFORM java.util.stream.Sink$ChainedInt [null, module java.base, Thread[main,5,main], loaded=true]
[Byte Buddy] INSTALL net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport@2a54a73f on sun.instrument.InstrumentationImpl@16a0ee18
Java version: 14.0.2
[Byte Buddy] TRANSFORM java.util.Properties [null, module java.base, Thread[main,5,main], loaded=true]
[+] public java.lang.String java.util.Properties.getProperty(java.lang.String)
[-] public java.lang.String java.util.Properties.getProperty(java.lang.String)
Java version: 14.0.2
[Byte Buddy] TRANSFORM java.util.IdentityHashMap$IdentityHashMapIterator [null, module java.base, Thread[main,5,main], loaded=false]
[Byte Buddy] TRANSFORM java.util.IdentityHashMap$KeyIterator [null, module java.base, Thread[main,5,main], loaded=false]
[+] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[-] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[+] public java.lang.Object java.util.IdentityHashMap$KeyIterator.next()
[+] protected int java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex()
[-] protected int java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex()
[-] public java.lang.Object java.util.IdentityHashMap$KeyIterator.next()
[+] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[-] public boolean java.util.IdentityHashMap$IdentityHashMapIterator.hasNext()
[Byte Buddy] TRANSFORM java.lang.Shutdown [null, module java.base, Thread[DestroyJavaVM,5,main], loaded=false]
[Byte Buddy] TRANSFORM java.lang.Shutdown$Lock [null, module java.base, Thread[DestroyJavaVM,5,main], loaded=false]
[+] static void java.lang.Shutdown.shutdown()
[+] private static void java.lang.Shutdown.runHooks()
[-] private static void java.lang.Shutdown.runHooks()
[-] static void java.lang.Shutdown.shutdown()

Please note how the first call of System.getProperties().getProperty("java.version") does not yield advice logging, but the second call after retransformation does.

---

Update after taking a look at your GitHub repository:

Am I understanding correctly? Module launcher tries to dynamically attach module agent to another, already running JVM. This looks complicated. Did you try with starting the other JVM with a -javaagent:/path/to/agent.jar parameter? You can still try the other strategy later. But either way, please note that your agent classes Agent and CompleteSTE will not be on the boot classpath like this.

Given the fact that advice code will be inlined into target classes (also bootstrap classes), it means that the bootstrap classes need to be able to find all classes referenced by the advice code on the boot classpath. There are two ways to achieve that:

1. Add -Xbootclasspath/a:/path/to/agent.jar to the target JVM command line in addition to -javaagent:/path/to/agent.jar. This of course only works, if you have influence on the target JVM's command line. Dynamic attachment to any running JVM does not work that way, because you are too late to specify a boot classpath option.

2. Partition the actual agent into a "springboard agent" and another JAR containing classes referenced by your advice code. The additional JAR can be packaged inside the agent JAR as a resource or reside somewhere on the filesystem, depending on how universal your solution ought to be. The springboard agent would

• optionally unpack the additional JAR to a temporary location (if nested inside the springboard agent),
• dynamically add the additional JAR to the boot classpath by calling method Instrumentation::appendToBootstrapClassLoaderSearch(JarFile),
• make sure not to reference any of the classes from the additional JAR directly, but if at all, only via reflection after the JAR is on the boot classpath already. Think Class.forName(..).getMethod(..).invoke(..).

BTW, if the classes referenced by the Byte Buddy (BB) advice use the BB API themselves, you also need to put BB itself on the boot classpath. All of this is far from trivial, so you want to try and avoid that. I went through all of this when trying to figure out how to best implement my special-purpose mocking tool Sarek.

---

Update 2: I mavenised and massively restructured the OP's original repository in this GitHub fork.

Answer 2

The enableBootstrapInjection method was replaced with a generic API that allows several injection strategies. This previous strategy is still available via an InjectionStrategy that uses instrumentation. This is mainly a reaction to the current flavors of Unsafe since the JVM is shutting down on internal APIs.

As you said, you need to refine your ignore matcher to allow some classes from the boot loader. The more classes you ignore by name, the better. Advice is the right approach for such classes as you can only add code but not change the shape of any class.

As it was mentioned, it is not necessary to put Byte Buddy on the boot path. As a matter of fact, advice methods are just templates, their code will be copy-pasted into the targeted methods. As a consequence, you do not have access to any fields or other methods within these advice classes.