Ember upgrade breaks one test, and only in Safari

Question

I'm trying to catch a Heisenbug.

I'm updating our project from Ember CLI 0.2.0 and Ember 1.10.0 to Ember CLI 0.2.3 and Ember 1.11.1. This has been a pretty pain-free process, but I have exactly one test which now fails in Safari (7.1.5) only. It passes in PhantomJS, Chrome and Firefox.

Annoyingly, the test only fails when the test run is initiated by Testem (that is, when a change in the code triggers an auto-update test run). If I initiate tests from inside the Qunit web interface, it passes. Both of these things are true regardless of the test grouping. The feature being tested works just fine when run manually in the browser.

It's an integration test and verifies that when a value is changed in an input, the input updates with the value returned from the server. In the tests, the "server" is a Pretender instance. Here's what the test itself looks like:

test('Editing allocation cell', function() {
  visit('/district/periods');

  fillIn(SELECTORS.definitionRowInput(1,0), '100');
  triggerEvent(SELECTORS.definitionRowInput(1,0), 'focusout');
  // The triggerEvent should be tripping the focusOut event on a particular
  // Ember.Textfield subclass, which subsequently leads to a POST request to
  // the server. On Safari, however, the focusOut event isn't being called here.
  // It is called elsewhere in the app, and it works in production.
  // Things that also don't work: keyEvent(element, 'keypress', 16) (a tab), 
  // sending 'blur', sending 'focus-out'.
  // 'focus-out' also fails in Firefox, 'blur' and tab fail in all 4 envs

  andThen(function() {
    equal($(SELECTORS.definitionRowInput(1,0)).val(), '90', 'The updated input takes the return value from the server (even if it is different from input)');
    equal($(SELECTORS.gradeTotal(2)).text(), '120', 'Grade total updates with the new sum');
  });
});

Note the second andThen() block: by sending focusout to the control, we should be prompting code in the backing component to update the data back to the server. The other browsers do this - I put a console.log() in the Pretender responder to verify it - but Safari doesn't. I'm guessing it's not responding properly to the focusout event.

Considering that this test passes in a branch which only differs by the Ember CLI update... what's likely to have changed to make this break?

ETA: I've individually rolled back all the libraries updated in this update and the test continues to fail. The only change which seems to work is rolling back Ember itself. It breaks in 1.11.0 in the same way as 1.11.1, so the change is between 1.10.0 and 1.11.0. (That only leaves me ~600 commits to sift through...)

ETA2: I've narrowed the scope to between 1.11.0-beta.5 and 1.11.0. I'm trying to use bower link to use local builds of ember but the ember builds have been unreliable so far in running the tests and the relationship of those two tags isn't one which leads to effective bisecting.

Answer 1

I can't help you with actually running your tests, but flattening the history isn't that difficult. There are 49 patches between the tags you mention. A stream of git cherry-pick commands gives the branch I've uploaded at https://github.com/rdebath/test/tree/ember.js

Each one of the commits on that branch (after the v1.11.0-beta.5 tag) comes from a 'nice' route between the tags you mentioned. The commit hashes are all different (obviously) but the final tree hash is the same as v1.11.0 so this should be a good path for git bisect.

The build problems can also be avoided, for example, I would suggest using a bisect to find the patch that causes them and git rebase -i that patch as late as you can. That should put a problem right next to it's fix; but it's probably not a good idea to "squash" those commits as you want to be able to relate everything back to the real tree.

To help choose the list of commits I used the command:

git log --graph --decorate --oneline --date-order --all

With that the "nice" path is reasonably obvious.