sbt and scala.js (with Node.js) can't run with a local .js dependency due to "TypeError: undefined is not a function"

Question

I need help with an error when I run using sbt, scala.js, a local bit of javascript code on Node.js.

[info] Running net.walend.graph.results.PlotTime
Hello from scala
[error] /Users/dwalend/projects/ScalaGraphMinimizer/toGhPages/target/scala-2.11/toghpages-fastopt.js:1854
[error]   $g["hello"]();
[error]              ^
[error] TypeError: undefined is not a function
[error]     at $c_Lnet_walend_graph_results_PlotTime$.main__V (/Users/dwalend/projects/ScalaGraphMinimizer/toGhPages/target/scala-2.11/toghpages-fastopt.js:1854:14)
[error]     at $c_Lnet_walend_graph_results_PlotTime$.$$js$exported$meth$main__O (/Users/dwalend/projects/ScalaGraphMinimizer/toGhPages/target/scala-2.11/toghpages-fastopt.js:1861:8)
[error]     at $c_Lnet_walend_graph_results_PlotTime$.main (/Users/dwalend/projects/ScalaGraphMinimizer/toGhPages/target/scala-2.11/toghpages-fastopt.js:1864:15)
[error]     at Object.<anonymous> (/Users/dwalend/projects/ScalaGraphMinimizer/toGhPages/target/scala-2.11/toghpages-launcher.js:2:107)
[error]     at Module._compile (module.js:460:26)
[error]     at Object.Module._extensions..js (module.js:478:10)
[error]     at Module.load (module.js:355:32)
[error]     at Function.Module._load (module.js:310:12)
[error]     at Module.require (module.js:365:17)
[error]     at require (module.js:384:17)
org.scalajs.jsenv.ExternalJSEnv$NonZeroExitException: node.js exited with code 1
      at org.scalajs.jsenv.ExternalJSEnv$AbstractExtRunner.waitForVM(ExternalJSEnv.scala:96)
      at org.scalajs.jsenv.ExternalJSEnv$ExtRunner.run(ExternalJSEnv.scala:143)
      at org.scalajs.sbtplugin.ScalaJSPluginInternal$.org$scalajs$sbtplugin$ScalaJSPluginInternal$$jsRun(ScalaJSPluginInternal.scala:479)
      at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$45$$anonfun$apply$27$$anonfun$apply$28.apply(ScalaJSPluginInternal.scala:539)
      at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$45$$anonfun$apply$27$$anonfun$apply$28.apply(ScalaJSPluginInternal.scala:533)
      at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)

I'm most suspicious of my build.sbt. (It's in a subproject that has no scala.js in it.) I think I have something out of joint, but don't know what other settings to try.

scalaVersion := "2.11.7"

scalacOptions ++= Seq("-unchecked", "-deprecation","-feature")

libraryDependencies ++= Seq(
  "org.scala-js" %%% "scalajs-dom" % "0.8.1"
)

//don't need phantomjs . //jsDependencies += RuntimeDOM

jsDependencies += "org.webjars" % "d3js" % "3.5.5-1" / "d3.min.js"

jsDependencies += ProvidedJS / "algorithmTime.js"

scalaJSStage in Global := FastOptStage

persistLauncher := true

I'm not able to even get a "hello" out of algorthmTime.js with Node.js.

function hello() {
  console.log("hello from js")
}

The main() in Scala pretty trim:

object PlotTime extends js.JSApp {

def main(): Unit = {
  println("Hello from scala")
  global.hello()

  val png = global.dataToPng("benchmark/results/v0.1.2/dijkstra.csv")
  println(png)
}

}

Before trying Node.js I got a bit further using phantom.js and Rhino. sbt run gets into my local javascript code and stalls inside of d3 with

[info] Running net.walend.graph.results.PlotTime
Hello from scala
hello from js
org.mozilla.javascript.EcmaError: TypeError: Cannot call method "querySelector" of undefined (/Users/dwalend/.ivy2/cache/org.webjars/d3js/jars/d3js-3.5.5-1.jar#META-INF/resources/webjars/d3js/3.5.5/d3.min.js#3)
  at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3701)
  at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3679)
  at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3707)
  at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3726)
  at org.mozilla.javascript.ScriptRuntime.undefCallError(ScriptRuntime.java:3743)
  at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2269)
  at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2262)
  at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1317)
  at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
  at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
  at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
  at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3102)
  at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120)
  at org.mozilla.javascript.Context.evaluateString(Context.java:1078)
  at org.scalajs.jsenv.rhino.package$ContextOps$.evaluateFile$extension(package.scala:21)
  at org.scalajs.jsenv.rhino.RhinoJSEnv.org$scalajs$jsenv$rhino$RhinoJSEnv$$internalRunJS(RhinoJSEnv.scala:157)
  at org.scalajs.jsenv.rhino.RhinoJSEnv$Runner.run(RhinoJSEnv.scala:62)
  at org.scalajs.sbtplugin.ScalaJSPluginInternal$.org$scalajs$sbtplugin$ScalaJSPluginInternal$$jsRun(ScalaJSPluginInternal.scala:479)
  at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$45$$anonfun$apply$27$$anonfun$apply$28.apply(ScalaJSPluginInternal.scala:539)
  at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$45$$anonfun$apply$27$$anonfun$apply$28.apply(ScalaJSPluginInternal.scala:533)
  at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
[trace] Stack trace suppressed: run last toGhPages/compile:run for the full output.
java.lang.RuntimeException: Exception while running JS code: TypeError: Cannot call method "querySelector" of undefined (/Users/dwalend/.ivy2/cache/org.webjars/d3js/jars/d3js-3.5.5-1.jar#META-INF/resources/webjars/d3js/3.5.5/d3.min.js#3)
  at scala.sys.package$.error(package.scala:27)
  at org.scalajs.jsenv.rhino.RhinoJSEnv.org$scalajs$jsenv$rhino$RhinoJSEnv$$internalRunJS(RhinoJSEnv.scala:173)
  at org.scalajs.jsenv.rhino.RhinoJSEnv$Runner.run(RhinoJSEnv.scala:62)
  at org.scalajs.sbtplugin.ScalaJSPluginInternal$.org$scalajs$sbtplugin$ScalaJSPluginInternal$$jsRun(ScalaJSPluginInternal.scala:479)
  at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$45$$anonfun$apply$27$$anonfun$apply$28.apply(ScalaJSPluginInternal.scala:539)
  at org.scalajs.sbtplugin.ScalaJSPluginInternal$$anonfun$45$$anonfun$apply$27$$anonfun$apply$28.apply(ScalaJSPluginInternal.scala:533)
  at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)

This error suggests my code is doing what it is supposed to. However, the internet wisdom says chasing in Rhino "querySelector" is a dead end and Node.js is a better choice.

I suspect I'm missing some sbt switch in the system, but don't know what else to look for.

I also don't see how it should work. I'm new to javascript, but I don't see how any one of these javascript files depends on any other in any of the produced files. (The examples on scala.js's tutorial link everything together using script tags in an index.html page.)

> tree toGhPages/target/scala-2.11/
toGhPages/target/scala-2.11/
├── classes
│   ├── JS_DEPENDENCIES
│   ├── algorithmTime.js
│   └── net
│       └── walend
│           └── graph
│               └── results
│                   ├── PlotTime$.class
│                   ├── PlotTime$.sjsir
│                   └── PlotTime.class
├── toghpages-fastopt.js
├── toghpages-fastopt.js.map
└── toghpages-jsdeps.js

The big picture: I'm attempting to use sbt, scala.js, and d3 to create performance charts for a scala graph algorithm library. The first cut of charts look promising, but github doesn't support javascript on README.md pages. For that I'll need a simple image. I want to learn more about both scala.js and d3 which attracted me to this approach.

Answer 1

Quickfix

In order to work in Node.js, do not properly declare the members you want to be visible (i.e. no var or named function):

hello = function() {
  console.log("hello from js")
};

This is a terrible hack, but will solve the inclusion problems for algorithmTime.js. "Proper" solution at the end.

Background

Composing different JavaScript files in a general is hard, since there exists no standardized way of doing so. Traditional HTML-include tags just have the semantics of concatenating all the code. This is the semantics we try to emulate in the Scala.js runners.

However, Node.js uses the CommonJS module system. In that system, a library explicitly exports members and the using site puts them into a namespace. This avoids naming collisions.

Example:

// Library (foo.js)
exports.foo = function() { return 1; };

// Using code
var lib = require("foo.js");
lib.foo() // returns 1

This allows the library to declare local values without leaking them into the caller. (Note aside: Although we have a function called require here, this is not RequireJS).

However, in the Scala.js runners, where we are expected to "just include" foo.js, this poses a challenge. What name should we use for the result of the require call? This is what commonJSName is means (see below for example).

If commonJSName for a given dependency is not set, in the Node.js runner, we will just emit

require(<name.js>);

without assigning it to anything. (Why not just dump the file you say? Goodbye reasonable stacktraces).

This has a very interesting effect in Node.js. Consider the following file (bar.js):

var a = 1;
b = 2;

Now we do:

require("bar.js")
console.log(a); // undefined
console.log(b); // 2

It seems that the b leaks into the global context whereas a does not. This is why the quickfix works.

Solutions

For a better solution, you have two choices:

1. Commit to Node.js, write your library specific to its module system
2. Autodetect the environment you are included in and adapt dynamically (many JS libraries do this)

Solution 1

modules.exports = function() {
  console.log("hello from js")
};

Add commonJSName to your dependency:

jsDependencies += ProvidedJS / "algorithmTime.js" commonJSName "hello"

This will fail miserably in anything but Node.js for two reasons:

1. The JS VM might not support CommonJS style includes
2. Overriding the full exports namespace like that is not standard CommonJS but specific to Node.js (IIRC).

Solution 2

Autodetect:

var hello = {};

// Scope to prevent leakage
(function(exp) {
  hello.hello = function() {
    console.log("hello from js");
  }
})(exports ? exports : hello);

You will also need to set commonJSName in this case.

Further, you might already suspect from the code, that this requires you to have an additional indirection, since CommonJS requires the top-level export to be an object (IIRC). Therefore you need to adapt your Scala.js code:

global.hello.hello();

However, if your library exports multiple symbols, this is probably a good idea anyway. Further, this is likely to work in most JS environments (and should work in the three environments we provide with Scala.js).

Epilogue

We (the Scala.js team) are very unhappy about this situation since we believe that including JS libraries should be just as easy as depending on other Scala and/or Java libraries in JVM land. However, we have not found a better solution to this short of supporting every inclusion style, which is a huge design, engineering and certainly maintenance effort (what if a system changes or a new system comes up?).

Related discussions: #457 and #706.