How to make custom task avoid redoing work if the input files are unchanged?

Question

I have a multi-project setup for a game. There is a very specific sub project called 'resources' that only contains files like images, sounds and texfiles to be packed into a jar.

I have a custom task that processes images and packs them. Inside 'src/main' I'm using a folder 'preprocess' where images should go and an 'unmanaged' folder where everything else goes. By running my task all images in 'preprocess' gets packed and output to 'resources' and everything in 'unmanaged' is copied as is.

val texturePacker = TaskKey[Unit]("texture-packer", "Runs libgdx's Texture Packer")

val texturePackerTask = texturePacker := {
  println("Packaging textures...")
  val inputDir = file("resources/src/main/preprocess")
  val outputDir = file("resources/src/main/resources")

  val folders = inputDir.asFile.listFiles filter (_.isDirectory)

  println("Sub-Folders:" + folders.mkString(", "))

  // Run Texture Packer
  for (subfolder <- folders) {
    println("Building assets for:" + subfolder)
    val args = Array(subfolder.toString, outputDir.toString, subfolder.getName)
    com.badlogic.gdx.tools.imagepacker.TexturePacker2.main(args)
  }

  // Copy unmanaged resources
  IO.copyDirectory(file("resources/src/main/unmanaged"), file("resources/src/main/resources"))
}

And then inside the settings the 'resources' project:

...
packageBin in Compile <<= packageBin in Compile dependsOn(texturePacker)
...

The other sub projects have a dependency on packageBin associated with their run. That way whenever I run the project I get the most up to date state of resources. I don't want it to be on demand. The problem is that it takes a long time to process for every run. I know SBT supports caching SBT FAQ but I don't understand how to adapt it to my task.

How can I make my custom task avoid redoing the work if the files in a subfolder from the folders list were not modified?

Answer 1

Solution

Here is a solution that might suite you. However, I don't fully understand how FileFunction.cached works (more information after the code), so this is probably not the best possible solution:

val testCache = TaskKey[Unit]("test-cache", "Test SBT's cache")

val testCacheTask = testCache := {
  println("Testing cache ...")

  val inputDir = file("test/src")   /* Take direct subdirectories from here */
  val outputDir = file("test/dest") /* Create archives here */
  val cacheDir = file("test/cache") /* Store cache information here */

  /* Get all direct subdirectories of inputDir */
  val folders = inputDir.asFile.listFiles.filter{_.isDirectory}

  folders.foreach{folder =>
    /* Get all files in the folder (not recursively) */
    val files = folder.listFiles.toSet

    /* Wrap actual function in a function that provides basic caching
     * functionalities.
     */
    val cachedFun =
      FileFunction.cached(cacheDir / folder.name,
                          FilesInfo.lastModified, /* inStyle */
                          FilesInfo.exists)       /* outStyle */
                         {(inFiles: Set[File]) =>

        createJarFromFolder(folder,
                            inFiles,
                            outputDir / (folder.name + ".jar"))
      }

    /* Call wrapped function with files in the current folder */
    cachedFun(files)
  }
}

/* Creates a JAR archive with all files (this time recursively) in
 * the given folder.
 */
val createJarFromFolder = (folder: File, inFiles: Set[File], outJar: File) => {
  println("At least one of the %d files in %s have changed. Creating %s."
          .format(inFiles.size, folder, outJar))

  /* Replace this by your actual operation */
  val cmdSeq = Seq("jar", "cf", outJar.toString, "-C" , folder + "/", ".")
  println("cmdSeq = " + cmdSeq)
  println("jar: " + cmdSeq.!!)

  Set(outJar)
}

Notes

• My understanding of cached is, that it checks the inFiles for modifications, and that it invokes the actual operation if one of the files in the set changed. The exact meaning of changed is determined by the inStyle argument to cached.

• It would be nice to directly pass a directory to cached, such that the actual operation is performed if anything in that directory changes. However, I doubt that it is currently possible.

• I don't quite get what the behaviour with respect to the set of files returned by the actual operation is (here:Set(outJar)). I assume that the outStyle argument to cached is related to this, and I expected createJarFromFolder to be called whenever the JAR does not exist (regardless of changes to the input files), but that doesn't seem to be the case. That is, if you delete a JAR file but not change one of the files in the corresponding directory, the JAR will not be recreated.

• The code is somewhat dodgy, because it only considers the files that are at the top of a particular folder when it comes to deciding if changes occurred in that folder. You probably want to make that recursive.

Epilogue

I'd love to see a better way of using SBT's caching feature. Should you get more information, e.g., from the mailing list, please post them here.