Register custom URLStreamHandler in Spring web application (Tomcat)

Question

I'm trying to register a custom URLStreamHandler to handle requests to Amazon S3 URLs in a generic fashion. The implementation of the handler looks pretty much like S3-URLStreamHandler (github). I did not put my Handler class into the sun.net.www.protocol.s3 package, but used a custom the package com.github.dpr.protocol.s3. To make Java pickup this package, I provided the system property -Djava.protocol.handler.pkgs="com.github.dpr.protocol" following the documentation of the URL class. However, if I try to process a s3-URL like s3://my.bucket/some-awesome-file.txt, I get a MalformedURLException:

Caused by: java.net.MalformedURLException: unknown protocol: s3
    at java.net.URL.<init>(URL.java:600)
    at java.net.URL.<init>(URL.java:490)
    at java.net.URL.<init>(URL.java:439)
    at java.net.URI.toURL(URI.java:1089)
    ...

My application is a Spring based web application, that currently runs in tomcat, but should not be cluttered with any knowledge about the underlying application container.

I already debugged the respective code and found that my URLStreamHandler can't be initialized as the classloader used to load the class doesn't know it. This is the respective code from java.net.URL (jdk 1.8.0_92):

1174: try {
1175:   cls = Class.forName(clsName);
1176: } catch (ClassNotFoundException e) {
1177:   ClassLoader cl = ClassLoader.getSystemClassLoader();
1178:   if (cl != null) {
1179:     cls = cl.loadClass(clsName);
1180:   }
1181: }

The class loader of the java.net.URL class (used by Class.forName) is null indicating the bootstrap class loader and the system class loader doesn't know my class. If I put a break point in there and try to load my handler class using the current thread's class loader, it works fine. That is my class is apparently there and is in the application's class path, but Java uses the "wrong" class loaders to lookup the handler.

I'm aware of this question on SOF, but I must not register a custom URLStreamHandlerFactory as tomcat registers it's own factory (org.apache.naming.resources.DirContextURLStreamHandlerFactory) on application start and there must only be one factory registered for a single JVM. Tomcat's DirContextURLStreamHandlerFactory allows to add user factories, that could be used to handle custom protocols, but I don't want to add a dependency to Tomcat in my application code, as the application should run in different containers.

Is there any way to register a handler for custom URLs in a container independent fashion?

---

UPDATE 2017-01-25:
I gave the different options @Nicolas Filotto proposed a try:

Option 1 - Set custom URLStreamHandlerFactory using reflection
This option works as expected. But by using reflection it introduces a tight dependency to the inner workings of the java.net.URL class. Luckily Oracle is not too eager with fixing convenience issues in the basic java classes - actually this related bug report is open for almost 14 years (great work Sun/Oracle) - and this can easily be unit tested.

Option 2 - Put handler JAR into {JAVA_HOME}/jre/lib/ext
This option works as well. But only adding the handler jar as system extension won't do the trick - of course. One will need to add all dependencies of the handler as well. As these classes are visible to all applications using this Java installation, this may lead to undesired effects due to different versions of the same library in the classpath.

Option 3 - Put handler JAR into {CATALINA_HOME}/lib
This doesn't work. According to Tomcat's classloader documentation resources put into this directory will be loaded using Tomcat's Common classloader. This classloader will not be used by java.net.URL to lookup the protocol handler.

Given these options I'll stay with the reflection variant. All of the options are not very nice, but at least the first one can easily be tested and does not require any deployment logic. However I adapted the code slightly to use the same lock object for synchronization as java.net.URL does:

public static void registerFactory() throws Exception  {
  final Field factoryField = URL.class.getDeclaredField("factory");
  factoryField.setAccessible(true);
  final Field lockField = URL.class.getDeclaredField("streamHandlerLock");
  lockField.setAccessible(true);

  // use same lock as in java.net.URL.setURLStreamHandlerFactory
  synchronized (lockField.get(null)) {
    final URLStreamHandlerFactory urlStreamHandlerFactory = (URLStreamHandlerFactory) factoryField.get(null);
    // Reset the value to prevent Error due to a factory already defined
    factoryField.set(null, null);
    URL.setURLStreamHandlerFactory(new AmazonS3UrlStreamHandlerFactory(urlStreamHandlerFactory));
  }
}

Answer 1

You could either:

1. Use a decorator

One way to set your custom URLStreamHandlerFactory, could be to use a decorator of type URLStreamHandlerFactory in order to wrap the URLStreamHandlerFactory that may have already been defined (by tomcat in this case). The tricky part is the fact that you need to use reflection (which is quite hacky) to get and reset the current factory potentially defined.

Here is the pseudo-code of your decorator:

public class S3URLStreamHandlerFactory implements URLStreamHandlerFactory {

    // The wrapped URLStreamHandlerFactory's instance
    private final Optional<URLStreamHandlerFactory> delegate;

    /**
     * Used in case there is no existing URLStreamHandlerFactory defined
     */
    public S3URLStreamHandlerFactory() {
        this(null);
    }

    /**
     * Used in case there is an existing URLStreamHandlerFactory defined
     */
    public S3URLStreamHandlerFactory(final URLStreamHandlerFactory delegate) {
        this.delegate = Optional.ofNullable(delegate);
    }

    @Override
    public URLStreamHandler createURLStreamHandler(final String protocol) {
        if ("s3".equals(protocol)) {
            return // my S3 URLStreamHandler;
        }
        // It is not the s3 protocol so we delegate it to the wrapped 
        // URLStreamHandlerFactory
        return delegate.map(factory -> factory.createURLStreamHandler(protocol))
            .orElse(null);
    }
}

Here is the code to define it:

// Retrieve the field "factory" of the class URL that store the 
// URLStreamHandlerFactory used
Field factoryField = URL.class.getDeclaredField("factory");
// It is a package protected field so we need to make it accessible
factoryField.setAccessible(true);
// Get the current value
URLStreamHandlerFactory urlStreamHandlerFactory 
    = (URLStreamHandlerFactory) factoryField.get(null);
if (urlStreamHandlerFactory == null) {
    // No factory has been defined so far so we set the custom one
    URL.setURLStreamHandlerFactory(new S3URLStreamHandlerFactory());
} else {
    // Retrieve the field "streamHandlerLock" of the class URL that
    // is the lock used to synchronize access to the protocol handlers 
    Field lockField = URL.class.getDeclaredField("streamHandlerLock");
    // It is a private field so we need to make it accessible
    lockField.setAccessible(true);
    // Use the same lock to reset the factory
    synchronized (lockField.get(null)) {
        // Reset the value to prevent Error due to a factory already defined
        factoryField.set(null, null);
        // Set our custom factory and wrap the current one into it
        URL.setURLStreamHandlerFactory(
            new S3URLStreamHandlerFactory(urlStreamHandlerFactory)
        );
    }
}

NB: Starting for Java 9, you will need to add --add-opens java.base/java.net=myModuleName to your launch command to allow deep reflection on the package java.net that includes the class URL from your module myModuleName otherwise calling setAccessible(true) will raise a RuntimeException.

---

2. Deploy it as an extension

Another way that should avoid this ClassLoder issue could be to move your custom URLStreamHandler into a dedicated jar and deploy it in ${JAVA-HOME}/jre/lib/ext as an installed extension (with all its dependencies) such that it will be available in the Extension ClassLoader, this way your class will be defined in a ClassLoader high enough in the hierarchy to be seen.