SBT generate code using project defined generator

Question

I'd like to compile a project which contains a java source generator and then compile the generated code within a single project. I.e: compile Generator.scala, run Generator.generate(outputDir), compile outputDir, package into a jar. I'm trying this:

sourceGenerators in Compile <+= sourceManaged in Compile map { out =>
    Generator.generate(out / "generated")
}

but sbt complains

[error] Build.scala:1: object example is not a member of package org
[error] import org.example.Generator

Basically, sbt doesn't see Generator defined in the project it compiles. Is it possible to do it my way with sbt?

Answer 1

So, after digging on this a bit, I have come up with a solution. First, you need to break your project into two sub projects. gen has all the source that includes your generator code. use depends on gen and uses the generator.

    import sbt._
    import Keys._
    import java.io.{ File ⇒ JFile, FileOutputStream }

    object OverallBuild extends Build {

      lazy val root = Project(id = "overall", base = file(".")).aggregate(gen, use)

      lazy val gen = Project(id = "generate", base = file("gen"))

      val myCodeGenerator = TaskKey[Seq[File]]("mycode-generate", "Generate My Awesome Code")

      lazy val use = Project(id = "use", base = file("use"),
        settings = Defaults.defaultSettings ++ Seq(

          sourceGenerators in Compile <+= (myCodeGenerator in Compile),

          myCodeGenerator in Compile <<=
            (javaSource in Compile, dependencyClasspath in Runtime in gen) map {

              (javaSource, cp) ⇒ runMyCodeGenerator(javaSource, cp.files)

            })).dependsOn(gen)

      def runMyCodeGenerator(javaSource: File, cp: Seq[File]): Seq[File] = {
        val mainClass = "com.yourcompany.myCodeGenerator"
        val tmp = JFile.createTempFile("sources", ".txt")
        val os = new FileOutputStream(tmp)

        try {
          val i = new Fork.ForkScala(mainClass).fork(None, Nil, cp,
            Seq(javaSource.toString),
            None,
            false,
            CustomOutput(os)).exitValue()

          if (i != 0) {
            error("Trouble with code generator")
          }
        } finally {
          os.close()
        }
        scala.io.Source.fromFile(tmp).getLines.map(f ⇒ file(f)).toList
      }
    }

In this case, I was generating .java files so I passed in javaSource to the generator.

It is important to not that when using sourceGenerators as we are here, the executed task must return a Seq[File] of all the files that were generated so that sbt can manage them. In this implementation, our generator outputs the full path file names to standard out and we save them to a temporary file.

As with all things Scala and surely SBT, you can do anything, just need to dig into it.

Answer 2

The project description is compiled when loading it. There is no way to directly call the new code generated at runtime. Unless I guess using some sort of reflection making sure there is no forking of the JVM and somehow having those classes loaded into the classloader.

The only way I can think of doing it is making a project inside your project definition.

root
 - src
 - project/
   - Build.scala // normal project definition
   - project/
     - Build.scala // inner most

In the inner most project definition you may be able to define the outer src as the src folder. That will get you a compiled version of the Generator available for the real project. Then in the normal project definition add an import to the Generator and use it as you were doing.

I'm pretty sure the inner most project will only be loaded and compiled once. You will need to have sbt reload the project definition if you make changes to the Generator. Exiting and reopening is the simplest/dumbest way of doing it but it may help testing. Lookup later smarter ways of reloading if it works.