Why does exception within frame get no notification in qUnit?

Question

I noticed that qUnit doesn't give any notice when an exception happens in a later part of the test. For example, running this in a test():

stop();
function myfun(ed) {
    console.log('resumed');
    start(); //Resume qunit
    ok(1,'entered qunit again');
    ok(ed.getContent()== 'expected content') // < causes exception, no getContent() yet.
}
R.tinymce.onAddEditor.add(myfun)

in an inner iframe on the page will cause an exception (TypeError: ed.getContent is not a function), but nothing in Qunit status area tells this. I see 0 failures.

(R being the inner iframe, using technique here: http://www.mattevanoff.com/2011/01/unit-testing-jquery-w-qunit/) Would I be correct in assuming this isn't the best way to go for testing sequences of UI interaction that cause certain results? Is it always better to use something like selenium, even for some mostly-javascript oriented frontend web-app tests?

As a side note, the Firefox console shows the console.log below the exception here, even though it happened first... why?

Answer 1

If you look into qUnit source code, there are two mechanisms handling exceptions. One is controlled by config.notrycatch setting and will wrap test setup, execution and teardown in try..catch blocks. This approach won't help much with exceptions thrown by asynchronous tests however, qUnit isn't the caller there. This is why there is an additional window.onerror handler controlled by Test.ignoreGlobalErrors setting. Both settings are false by default so that both kinds of exceptions are caught. In fact, the following code (essentially same as yours but without TinyMCE-specific parts) produces the expected results for me:

test("foo", function()
{
  stop();
  function myfun(ed)
  {
    start();
    ok(1, 'entered qunit again');
    throw "bar";
  }
  setTimeout(myfun, 1000);
});

I first see a passed tests with the message "entered qunit again" and then a failed one with the message: "uncaught exception: bar." As to why this doesn't work for you, I can see the following options:

1. Your qUnit copy is more than two years old, before qUnit issue 134 was fixed and a global exception handler added.
2. Your code is changing Test.ignoreGlobalErrors setting (unlikely).
3. There is an existing window.onerror handler that returns true and thus tells qUnit that the error has been handled. I checked whether TinyMCE adds one by default but it doesn't look like it does.
4. TinyMCE catches errors in event handlers when calling them. This is the logical thing to do when dealing with multiple callbacks, the usual approach is something like this:

for (var i = 0; i < callbacks.length; i++)
{
  try
  {
    callbacks[i]();
  }
  catch (e)
  {
    console.error(e);
  }
}

By redirecting all exceptions to console.error this makes sure that exceptions are still reported while all callbacks will be called even if one of them throws an exception. However, since the exception is handled jQuery can no longer catch it. Again, I checked whether TinyMCE implements this pattern - it doesn't look like it.

Update: Turns out there is a fifth option that I didn't think of: the exception is fired inside a frame and qUnit didn't set up its global error handler there (already because tracking frame creation is non-trivial, a new frame can be created any time). This should be easily fixed by adding the following code to the frame:

window.onerror = function()
{
  if (parent.onerror)
  {
    // Forward the call to the parent frame
    return parent.onerror.apply(parent, arguments);
  }
  else
    return false;
}

Concerning your side-note: the console object doesn't guarantee you any specific order in which messages appear. In fact, the code console.log("foo");throw "bar"; also shows the exception first, followed by the log message. This indicates that log messages are queued and handled delayed, probably for performance reasons. But you would need to look into the implementation of the console object in Firefox to be certain - this is an implementation detail.