Configure Karma to load pegjs with requirejs

Question

Trying to test a project using PegJS and requirejs. I have a couple of source files, implemented as AMD module (define) which loads through the require API. Below the directory structure:

js/
   somefile.js
   main.js
   parser.js
test/
   parser.spec.js

I've written a parser.js module to load a PegJS grammar file and use PegJS to create a peg parser:

define(function() {
  'use strict';

  var PEG = require('pegjs');
  var grammarFile = 'grammar.peg'

return {
  parse: function(fs, content, debug) {
    var grammar = fs.readFileSync(grammarFile, 'utf8').toString();
    // Build parser from grammar
    var parser = PEG.buildParser(grammar, { trace: debug });
    [...]

This works fine with a main.js executed on the command line with node. Now I want to test my project using karma, jasmine and PhantomJS. I have a karma.conf.js like this:

frameworks: ['jasmine', 'requirejs'],
files: [
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js',
],

Also have a require bootstrap file called test-main.js which is configured this way:

'use strict';

var allTestFiles = [];
var TEST_REGEXP = /(spec|test)\.js$/i;

// Get a list of all the test files to include
Object.keys(window.__karma__.files).forEach(function(file) {
  console.log(file);
  if (TEST_REGEXP.test(file)) {
    // Normalize paths to RequireJS module names.
    // If you require sub-dependencies of test files to be loaded as-is (requiring file extension)
    // then do not normalize the paths
    var normalizedTestModule = file.replace(/^\/base\/|\.js$/g, '');
    allTestFiles.push(file);
  }
});

require.config({
  // Karma serves files under /base, which is the basePath from your config file
  baseUrl: '/base/js',
  // dynamically load all test files
  deps: allTestFiles,
  // we have to kickoff jasmine, as it is asynchronous
  callback: window.__karma__.start
});

Now, when I launch my test (grunt karma), I got this error:

PhantomJS 1.9.8 (Linux 0.0.0) ERROR: Error{message: 'Module name "pegjs" has not been loaded yet for context: _. Use require([])

So I try to include pegjs in the files loaded by Karma this way karma.conf.js:

files: [
  { pattern: 'node_modules/pegjs/lib/**/*.js', included: true  },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

When I do this, I still get an error:

Error: Module name "utils/arrays" has not been loaded yet for context: _. Use require([])

Looking inside pegjs module, there is indeed an arrays.js file:

compiler/
compiler.js
grammar-error.js
parser.js
peg.js
utils/
  arrays.js
  classes.js
  objects.js

So trying to include arrays too:

files: [
  { pattern: 'node_modules/pegjs/lib/utils/arrays.js', included: true },
  { pattern: 'node_modules/pegjs/lib/**/*.js', included: true  },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

I get:

ReferenceError: Can't find variable: module
at /blabla/node_modules/pegjs/lib/utils/arrays.js:108

Because of:

108 module.exports = arrays;

So, intead of loading the npm module, I tried to load the bower module this way:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

And here you go again:

PhantomJS 1.9.8 (Linux 0.0.0) ERROR: Error{message: 'Module name "pegjs" has not been loaded yet for context: _. Use require([])

Also tried not to include pegjs in the karma generated web page:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: false },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

But it fails with:

PhantomJS 1.9.8 (Linux 0.0.0) ERROR: 'There is no timestamp for /base/bower_components/pegjs/peg-0.9.0!'

Tried to put the bower_component folder inside the js folder but no luck.

So I don't know were to go from here... Couldn't find anything relevant on Google or here. It seems to be a specific problem to requirejs/pegjs with karma... Any idea is welcome.

UPDATE following dan's answer:

So I switch from a synchronous require to a asynchronous require in parser.js:

define(['../bower_components/pegjs/peg-0.9.0'], function(PEG) {
  'use strict';

  var grammarFile = 'grammar.peg'

return {
  parse: function(fs, content, debug) {
    var grammar = fs.readFileSync(grammarFile, 'utf8').toString();
    // Build parser from grammar
    var parser = PEG.buildParser(grammar, { trace: debug });
    [...]

Tried to include the pegjs bower component in karma.conf.js:

{ pattern: 'bower_components/pegjs/peg-0.9.0.js', included: false },

or not include it:

{ pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },

But always get the same error:

Error: Script error for "/blabla/bower_components/pegjs/peg-0.9.0", needed by: /blabla/js/parser.js
http://requirejs.org/docs/errors.html#scripterror
at /blabla/node_modules/requirejs/require.js:140

Yes the file exists:

$ file /home/aa024149/share/logofrjs/bower_components/pegjs/peg-0.9.0.js 
/blabla/bower_components/pegjs/peg-0.9.0.js: ASCII text, with very long lines

UPDATE2: Finally understood and found an acceptable solution.

Answer 1

It sounds like you're loading pegjs via requirejs. If this is the case, pegjs should not be a file that is included. In your karma.conf.js, have you tried the following:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: false },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

The value for included indicates whether or not the webpage that the karma server generates should have a script tag for that file or not (See http://karma-runner.github.io/0.13/config/files.html). So your karma.config of:

files: [
  { pattern: 'bower_components/pegjs/peg-0.9.0.js', included: true },
  { pattern: './test/**/*.spec.js', included: false },
  { pattern: './js/**/*.js', included: false},
  './test/test-main.js'
],

will cause karma to generate a webpage with a head tag similar to:

<head>
  <script src="/base/bower_components/pegjs/peg-0.9.0.js"></script>
  <script src="/base/require.js"></script>
  <script src="/base/test/test-main.js"></script>
</head>

In my experience, I've seen a lot of behavior similar to this that was caused by marking my files as included: true. If there is a file which you are trying to load with requirejs, make sure that it is marked as included: false.

I believe that this marks it for a pre-processing job, but I'm not entirely sure why this makes such a difference.

Answer 2

As far as I know, Karma is a testing framework which will run your tests in the browser.

This isn't suitable for testing many node modules.

The browser doesn't have the facility to do this synchronously: var PEG = require('pegjs'). This is why it is asking you to use require([]) which you pass a callback to be executed when pegjs finishes loading.

Using the bower version of pegjs and ensuring it is loaded before require('pegjs') is called may help here. This would ensure that pegjs is already loaded for context _ (the default requirejs context, I presume).

It also can't load files from the file system with fs.readFileSync(grammarFile, 'utf8') so you will have to do it another way. You can ask Karma to host your peg grammar by placing it in the files array and then loading it using the requirejs text plugin.

If the module you are testing is aimed at running on node.js and not in the browser then it may be more suited to using a test framework that does not run the code in the browser, but runs it in node so you have all the node modules available to you. If you are aiming this at the browser, I would rewrite it to more specifically target the browser.