Changing the compiler's classpath from a Scala macro?

Question

Say we wanted to build a feature like require, which is known in many scripting languages, in Scala.

If we require a library like this...

require("some-library.jar")

... we would need a way to add this jar to the compiler's classpath when the require macro is executed. How can this be done?

Answer 1

Another interesting @kim-stebel question.

My first idea, which doesn't address your question, is that your compiler can customize the macro classpath with findMacroClassLoader. The REPL uses that thanks to @extempore.

That would be useful for an idiom like requiring("myfoo.jar") { mymacro }, perhaps.

Your question is whether you can update the compiler's classpath. That may be possible by casting down from your context universe to the compiler, for which isCompilerUniverse == true. Then you can platform.updateClassPath.

Update with code:

Here is something like that idea of using a special placeholder in the class path for the required macro.

The fancy way, mentioned below, would be to use a custom class path that can report all the required classes at that location. The old code mentioned below was for a virtual directory in the class path.

This quick and cheesy way uses the file system to unpack your jar and just asks the compiler to rescan it.

The placeholder has to be an actual dir due to a limitation in invalidateClassPathEntries, which wants to check if the canonical file path is on the class path.

package alacs

import scala.language.experimental.macros
import scala.reflect.macros.Context

import scala.sys.process._
import java.io.File

/** A require macro to dynamically fudge the compilation classpath.  */
object PathMaker {
  // special place to unpack required libs, must be on the initial classpath
  val entry = "required"

  // whether to report updated syms without overly verbose -verbose
  val talky = true

  def require(c: Context)(name: c.Expr[String]): c.Expr[Unit] = {
    import c.universe._
    val st = c.universe.asInstanceOf[scala.reflect.internal.SymbolTable]
    if (st.isCompilerUniverse) {
      val Literal(Constant(what: String)) = name.tree
      if (update(what)) {
        val global = st.asInstanceOf[scala.tools.nsc.Global]
        val (updated, _) = global invalidateClassPathEntries entry
        c.info(c.enclosingPosition, s"Updated symbols $updated", force = talky)
      } else {
        c.abort(c.enclosingPosition, s"Couldn't unpack '$what' into '$entry'")
      }
    }
    reify { () }
  }

  // figure out where name is, and update the special class path entry
  def update(name: String): Boolean = {
    // Process doesn't parse the command, it just splits on space,
    // something about working on Windows
    //val status = s"sh -c \"mkdir $entry ; ( cd $entry ; jar xf ../$name )\"".!

    // but Process can set cwd for you
    val command = s"jar xf ../$name"
    val status = Process(command, new File(entry)).!
    (status == 0)
  }
}

The required require API:

package alacs

import scala.language.experimental.macros

object Require {
  def require(name: String): Unit = macro PathMaker.require
}

Usage:

package sample

import alacs.Require._

/** Sample app requiring something not on the class path. */
object Test extends App {
  require("special.jar")
  import special._
  Console println Special(7, "seven")
}

Something packaged in special.jar

package special

case class Special(i: Int, s: String)

Tested this way:

rm -rf required
mkdir required
skalac pathmaker.scala
skalac -cp .:required require.scala sample.scala
skala -cp .:special.jar sample.Test

apm@mara:~/tmp/pathmaker$ . ./b
Unpack special.jar
sample.scala:8: Updated symbols List(package special)
  require("special.jar")
         ^
Special(7,seven)

The macro doesn't sneak it onto the runtime path, which is the scripty thing to do.

But I suppose the require macro could do handy things like conditionally grab different versions of a jar, with differing compile-time characteristics (constants, etc).

Update, just verifying that it's pretty wild:

  require("fast.jar")
  import constants._
  Console println speed

  require("slow.jar")
  Console println speed

where

package object constants {
  //final val speed = 55
  final val speed = 85
}

$ skalac -d fast.jar constants.scala

and running it with inlined constants

85
55

New caveat: this is my first macro, and I'm looking at invalidateClassPathEntries for another application, so I haven't explored limitations yet.

Update: one limitation is controlling when the macro is expanded. I wanted to show how something compiles against old api vs new api, and had to wrap the code in blocks to make sure symbols are available before needed:

require("oldfoo.jar")
locally {
  import foo._
  // something
  require("newfoo.jar")
  // try again
}

Old caveat: Sorry for the vague answer, I know you don't abide that; I'll try it out later, but meantime maybe someone will step forward with clarity.

Previously, I've hooked into the "platform" implementation for the compiler global, but that's hopefully overkill for this use case. At that point, you can do whatever you want with the classpath, but I think you want something more out-of-the-box.