Is there a way to hook into Webpack's AST to make it recognize a new module format?

Question

Short version:

How do we manipulate the AST for the final output bundle, as well as the AST for a file from inside a loader? In both cases, I'd like to manipulate an existing AST rather than what I'm doing which is to parse the sources and make a new AST. What I'm doing is slow, and I know that Webpack must already have made an AST so I want to avoid duplicating efforts.

Long version:

For example, suppose I have a bunch of files written in a format similar to (but not quite) AMD modules:

module({
  Foo: '/path/to/Foo',
  Bar: '/path/to/Bar',
  Baz: '/path/to/Baz',
}, function(imports) {
  console.log(imports) // {Foo:..., Bar:... Baz:...}
})

The difference is that it is called module instead of define, the dependencies argument is a map of import names to module paths instead of an array of module paths, and the module body function receives an import object with all requested imports instead of one argument per requested import.

The above is similar to the following in AMD format, with the same output:

define([
  '/path/to/Foo',
  '/path/to/Bar',
  '/path/to/Baz',
], function(Foo, Bar, Baz) {
  console.log({Foo, Bar, Baz}) // {Foo:..., Bar:... Baz:...}
})

What is the recommended way to hook into Webpack to make Webpack be able to understand the files (be able to know what dependencies the file has) in order to finally build a bundle with files that are written in this module() format?

I've already tried one approach: I made a custom loader that receives a file's source as a string, parses it and creates and AST, transform the AST, then outputs the code in AMD define() format, which Webpack understands.

However, I feel like this is slow, because if there are many files and if they are big, then parsing and making an AST from each files seem redundant, because I bet Webpack is already doing that to begin with. Is there some way to get the AST from Webpack and transform it before Webpack wants to scan it's dependencies, so that I can transform the AST into AMD format (or any recognized format for that matter), so that Webpack can finally work with the file? Is there another approach?

Answer 1

I think you will find that the loader is used during dependency parsing.

Basically the parser needs source code to be able to do its job. Therefore any import/require statement (a dependency) that is encountered during the current parse phase needs to: a. be resolved and: b. be loaded before it can be parsed. If you hook into the enhanced-resolve package's "resolve-step" you can console.log out the state transitions that the resolver transitions through typically culminating in the "create-module" plugins being fired.

Hook into "resolve-step":

compiler.plugin('after-resolvers', (compiler) => {
    compiler.resolvers.normal.plugin('resolve-step', function (type, request){
        console.log("resolve-step type:["+type+"], 
            path:["+request.path+"], request:["+request.request+"]");   
        });
});

Hook into "create-module":

compiler.plugin('compilation', (compilation, params) => {
    params.normalModuleFactory.plugin("create-module", (data) => {
        console.log('create-module: raw-request: '+data.rawRequest);
    }
}

Hope this helps.

Answer 2

I was looking for something like this, want to manipulate the ast but there is no example or useful documentation out there.

Googling for it just wasted 2 hours of my time but with the half completed and incomprehensible documentation I did come up with this (doesn't work though):

var acorn = require("acorn-dynamic-import").default;

function MyPlugin(options) {
  // Configure your plugin with options...
}

MyPlugin.prototype.apply = function(compiler) {
  compiler.plugin("compilation", function(compilation, data) {
    data.normalModuleFactory.plugin(
      "parser"
      , function(parser, options) {
        parser.plugin(
          [
            "statement if"
          ]
          ,(node)=>
            Object.assign(
              node
              ,{
                test:{
                  type:"Literal",
                  start: node.test.start,
                  end: node.test.start+4,
                  loc: {
                    start: {
                      line: 7,
                      column: 3
                    },
                    end: {
                      line: 7,
                      column: node.test.loc.start.column+4
                    }
                  },
                  range: [
                    node.test.range,
                    node.test.range+4
                  ],
                  value: true,
                  raw: "true"
                }
              }
            )
        ); 
      });
  });
};

module.exports = MyPlugin;

That would get me a node of an if statement, for other types of nodes you can look in Parser.js.

I try returning another node but creating a node is a lot of work and there doesn't seem to be an easy way to do this.

Why bother trying to do this in webpack anyway? The webpack contributors have made a nice product but nobody knows how it works because there is no documentation for many of it's features.

You're probably better off doing this as a babel plugin, that has well written and understandable documentation.

I got this working in about 20 minutes with babel:

module.exports = function ({ types: t }) {
    return {
        visitor: {
            IfStatement(path, file) {
        path.node.test = {
          "type": "BooleanLiteral",
          "start": path.node.test.start,
          "end": path.node.test.start+4,
          "loc": {
            "start": {
              "line": 7,
              "column": path.node.test.loc.start.column
            },
            "end": {
              "line": 7,
              "column": path.node.test.loc.start.column+4
            }
          },
          "value": true
        }
        // debugger;
                // path.unshiftContainer('body', t.expressionStatement(t.stringLiteral('use helloworld')));
      }
        }
  };
};

In webpack.config.js:

const path = require("path");
const webpack = require("webpack");

module.exports = {
  entry: "./src",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "[name].chunk.js"
  },
  module:{
    rules: [
      {
        test: /\.html$/,
        use: [{ loader: './plugin/templateLoader' }]
      },
      {
        test: /\.js$/,
        exclude: /node_modules/,
        use: {
          loader: 'babel-loader',
          options: {
            presets: ['env'],
            plugins: [require("./plugin/myBabelPlugin")]
          }
        },
      }
    ]
  }
}