Speed of scala's ant fsc task

Question

I have an ant file I use to compile my scala project. I'm using fsc which works wonders to avoid the 2~3 seconds my core 2 needs to just load the compiler.
My problem is: the ant fsc task does incur the same 2~3 seconds penalty, to the best of my knowledge. It is pretty annoying, because there is fsc specifically for this reason. It is even more annoying since it's really starting time and not processing time, so I have to wait 3 seconds on the fsc task even when there is nothing to re-compile. Each and every time, it becomes aggravating.

My investigation seems to show that most the time is spent reading scala-compiler.jar. Indeed, it makes sense the scalac and fsc tasks need this, since they run the compiler directly. Besides, removing scala-compiler.jar from the classpath of the ant task results, of course, in the task failing for lack of the dependency.
Logically speaking, the fsc task is only connecting to the compilation dæmon I suppose, so it shouldn't need that dependency, but I suppose it's about how it's implemented (the fsc task inherits from the scala task). Maybe it will come in a next version.

The solution I am looking at at the moment is, rewrite my fsc task as an apply task and invoke fsc manually. I won't have the delay then. It's just a pity to have to redo manually the work that the tasks included with scala were specifically written to handle.

Does anyone have experience with this problem? Is anything wrong in my analysis? And can you suggest a better solution than the one I am planning to implement?

For reference, here is what my task looks like (yes, it compiles an android project):

<target name="compile-scala" description="Compile scala files">
  <taskdef resource="scala/tools/ant/antlib.xml" classpath="${scala-library.jar}:${scala-compiler.jar}" />
  <mkdir dir="${out.classes.absolute.dir}" />
  <fsc encoding="utf-8" deprecation="on" destdir="${out.classes.absolute.dir}">
    <src>
      <dirset dir="." includes="src"/>
      <dirset dir="." includes="gen"/>
    </src>
    <classpath>
      <pathelement location="${android.jar}" />
      <fileset dir="${sdk.dir}/tools/lib" includes="*.jar" />
    </classpath>
  </fsc>
</target>

Edit: Here is what the task with apply looks like. It seems to work. It is however quite unsatisfying, so the question still holds.

<target name="fast-compile-scala"
    description="Compile scala files without loading the compiler inside ant">
  <mkdir dir="${out.classes.absolute.dir}" />
  <apply executable="fsc" failonerror="true" parallel="true">
    <arg value="-encoding" />
    <arg value="utf-8" />
    <arg value="-deprecation" />
    <arg value="-dependencyfile" />
    <arg value="${out.classes.absolute.dir}/${scala.deps}" />
    <arg value="-g:vars" />
    <arg value="-d" />
    <arg value="${out.classes.absolute.dir}" />
    <arg value="-make:transitivenocp" />
    <arg value="-classpath" />
    <arg value="${android.jar}:${out.classes.absolute.dir}" />
    <!-- <arg value="-optimize" /> -->
    <!-- <arg value="-verbose" /> -->
    <!-- <arg value="-explaintypes" /> -->
    <fileset dir="src" includes="**/*.scala" />
    <fileset dir="src" includes="**/*.java" />
    <fileset dir="gen" includes="**/*.java" />
  </apply>
</target>

Edit again: The above works, but I find it has a quite severe limitation. When the number of files grow, the above task is limited by the number of arguments and/or the length of the command line. Under linux or mac OS with a good shell you can go quite a way without hitting this wall, but under windows a couple dozen files is all it takes to prevent the above from working. Adding an option to specify files as relative paths instead of absolute gives some breath, but not much. Splitting files in several commands according to dependencies is not a realistic option for it is a quite heavy task and needs to be updated whenever the file structure changes. So while for small projects the above will mostly solve the issue, it won't help for any project of a respectable size...

Answer 1

In hope this will help you:

first case (fsc ant task)

When you start ant, you load a brand new JVM in which loads scala-compiler.jar. The jar is loaded because of the line <taskdef resource="scala/tools/ant/antlib.xml" classpath="${scala-library.jar}:${scala-compiler.jar}" /> not because of the classpath part.

second case (ant starts a command line)

As you outline, this is limited to simple projects with few files. Max command line length: windows xp -> 8191, linux/osx -> kernel value (usually >100000) By the way, you are spawning two JVM here (one for ant, the other for fsc). Perhaps it's faster in your tests as it use caches.

proposed solution

In maven there is the goal scala:cc (continuous compilation) which keep the compiler in memory as well as continuously scanning the files for modifications. If you don't want to change your builder to maven perhaps you could write a new ant task which does the same thing (fscc: a fast scala continuous compiler)