Webpack: Common chunks for code shared between Webworker and Web code?

Question

I have lots of code shared between web and web worker parts of my browser app.

How can I tell webpack to split my code up into common chunks so that the result is garanteed to work 100%?

The webworker code breaks (fails at runtime) after I tell webpack to generate the common chunks (which it does). Even after I fix the trivial "window not defined" error the worker just does nothing.

I believe this has to do with the webpack "target" option, which per default is set to "web". But I need "web" target because I don't have purely webworker code.

I also cannot do multiple webpack configs because I cannot do the common chunks thing with multiple configs...

What should I do?

If anybody is interested: I am trying build a minimal sized build for my app which includes the monaco editor (which provides the workers):

https://github.com/Microsoft/monaco-editor/blob/master/docs/integrate-esm.md

You can see here (at the bottom of the page) that the entry points consist of 1 main entry file + the workers.

Currently at least 6 MB is wasted because of duplicate code I am using and currently can not be split up because of this problem. That is a lot of wasted traffic.

Any ideas? :)

my webpack 4.1.1 config is basically:

module.exports = (env, options) => {
    const mode = options.mode;
    const isProduction = mode === 'production';
    const outDir = isProduction ? 'build/release' : 'build/debug';

    return {

        entry: {
            "app": "./src/main.tsx",
            "editor.worker": 'monaco-editor/esm/vs/editor/editor.worker.js',
            "ts.worker": 'monaco-editor/esm/vs/language/typescript/ts.worker.js'
        },
        output: {
            filename: "[name].bundle.js",
            path: `${__dirname}/${outDir}`,
            libraryTarget: 'umd',
            globalObject: 'this',
            library: 'app',
            umdNamedDefine: true
        },
        node: {
            fs: 'empty' 
        },
        devtool: isProduction ? undefined : "source-map",
        resolve: {
            extensions: [".ts", ".tsx", ".js", ".json"],
            alias: {
                "@components": path.resolve(__dirname, "src/components"),
                "@lib": path.resolve(__dirname, "src/lib"),
                "@common": path.resolve(__dirname, "src/common"),
                "@redux": path.resolve(__dirname, "src/redux"),
                "@services": path.resolve(__dirname, "src/services"),
                "@translations": path.resolve(__dirname, "src/translations"),
                "@serverApi": path.resolve(__dirname, "src/server-api")
            }
        },
        optimization: isProduction ? undefined : {
            splitChunks: {
                minSize: 30000,
                minChunks: 1,
                name: true,
                maxAsyncRequests: 100,
                maxInitialRequests: 100,
                cacheGroups: {
                    default: {
                        chunks: "all",
                        priority: -100,
                        test: (module) => {
                            const req = module.userRequest;
                            if (!req) return false;
                            return (!/node_modules[\\/]/.test(req));
                        },
                    },
                    vendor: {
                        chunks: "all",
                        test: (module) => {
                            const req = module.userRequest;
                            if (!req) return false;
                            if (!/[\\/]node_modules[\\/]/.test(req)) return false;
                            return true;
                        },
                        priority: 100,
                    }
                }
            },
        },
        module: {
            rules: [...(isProduction ? [] : [
                {
                    enforce: "pre", test: /\.js$/, loader: "source-map-loader",
                    exclude: [
                        /node_modules[\\/]monaco-editor/ 
                    ]
                }
            ]),
            {
                test: require.resolve('jquery.hotkeys'),
                use: 'imports-loader?jQuery=jquery'
            },
            {
                test: /\.tsx?$/,
                loader: "awesome-typescript-loader",
                options: {
                    configFileName: 'src/tsconfig.json',
                    getCustomTransformers: () => {
                        return {
                            before: [p => keysTransformer(p)]
                        };
                    }
                }
            },
            {
                test: /\.(css|sass|scss)$/,
                use: extractSass.extract({
                    use: [
                        {
                            loader: 'css-loader',
                            options: {
                                minimize: isProduction
                            }
                        },
                        {
                            loader: "postcss-loader",
                            options: {
                                plugins: () => [autoprefixer({
                                    browsers: [
                                        'last 3 version',
                                        'ie >= 10'
                                    ]
                                })]
                            }
                        },
                        { loader: "sass-loader" }
                    ],
                    fallback: "style-loader"
                })
            },
            {
                test: /node_modules[\/\\]font-awesome/,
                loader: 'file-loader',
                options: {
                    emitFile: false
                }
            },
            {
                test: { not: [{ test: /node_modules[\/\\]font-awesome/ }] },
                rules: [
                    {
                        test: { or: [/icomoon\.svg$/, /fonts[\/\\]seti\.svg$/] },
                        rules: [
                            { loader: 'file-loader?mimetype=image/svg+xml' },
                        ]
                    }, {
                        test: { not: [/icomoon\.svg$/, /fonts[\/\\]seti\.svg$/] },
                        rules: [
                            {
                                test: /\.svg(\?v=\d+\.\d+\.\d+)?$/,
                                use: {
                                    loader: 'svg-url-loader',
                                    options: {}
                                }
                            },
                        ]
                    },
                    {
                        test: /\.(png|jpg|gif)$/,
                        loader: 'url-loader'
                    },
                    { test: /\.woff(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?mimetype=application/font-woff" },
                    { test: /\.woff2(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?mimetype=application/font-woff" },
                    { test: /\.ttf(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?mimetype=application/octet-stream" },
                    { test: /\.eot(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader" },
                ]
            },

            ]
        },
        plugins: [
            new HardSourceWebpackPlugin({
                cacheDirectory: '../node_modules/.cache/hard-source/[confighash]', configHash: function (webpackConfig) {
                    return require('node-object-hash')({ sort: false }).hash(Object.assign({}, webpackConfig, { devServer: false }));
                },
                environmentHash: {
                    root: process.cwd(),
                    directories: [],
                    files: ['../package-lock.json'],
                }
            }),
            new webpack.ProvidePlugin({
                "window.$": "jquery"
            }),
            new CleanWebpackPlugin(outDir),
            extractSass,
            new HtmlWebpackPlugin({
                title: 'my title',
                filename: 'index.html',
                minify: isProduction ? {
                    collapseWhitespace: true,
                    collapseInlineTagWhitespace: true,
                    removeComments: true,
                    removeRedundantAttributes: true
                } : false,
                template: 'index_template.html',
                excludeChunks: ['ts.worker', "editor.worker"]
            }),
            new webpack.IgnorePlugin(/^((fs)|(path)|(os)|(crypto)|(source-map-support))$/, /vs[\\\/]language[\\\/]typescript[\\\/]lib/)
        ].concat(isProduction ? [new webpack.optimize.LimitChunkCountPlugin({
            maxChunks: 1
        })] : [])
    }
};

Answer 1

EDIT: Alright I wrote a webpack plugin based on everyone's knowledge just put together.

https://www.npmjs.com/package/worker-injector-generator-plugin

You can ignore the content below, and use the plugin or if you want to understand how the plugin came to be and do it by hand yourself (so you don't have to depend on my code) you can keep reading.

=====================================================

Alright after so much researching I figured out this solution, you need to create an injection file, for a simple case you need the https://github.com/webpack-contrib/copy-webpack-plugin as it works pretty well... so let's say your setup is:

entry: {
    "worker": ["./src/worker.ts"],
    "app": ["./src/index.tsx"],
  },

And you have setup your common plugins already let's say this example.

optimization: {
    splitChunks: {
      cacheGroups: {
        commons: {
          name: 'commons',
          chunks: 'initial',
          minChunks: 2
        },
      }
    }
  },

You need to now create an injection "Vanilla JS" which might look like this:

var base = location.protocol + "//" + location.host;
window = self;

self.importScripts(base + "/resources/commons.js", base + "/resources/worker.js");

Then you can add that alongside your worker, say in src/worker-injector.js

And using the copy plugin

new CopyPlugin([
      {
        from: "./src/worker-injector.js",
        to: path.resolve(__dirname, 'dist/[name].js'),
      },
    ]),

Make sure your output is set to umd.

output: {
    filename: '[name].js',
    path: path.resolve(__dirname, 'dist'),
    libraryTarget: "umd",
    globalObject: "this",
  }

This is nothing but a hack, but allows you to use everything as it is without having to do something as overblown.

If you need hashing (so that copy plugin doesn't work) functionality you would have to generate this file (rather than copying it), refer to this.

How to inject Webpack build hash to application code

For that you would have to create your own plugin which would generate the vanilla js file and consider the hash within itself, you would pass the urls that you want to load together, and it would attach the hash to them, this is more tricky but if you need hashes it should be straightforward to implement with your custom plugin.

Sadly so far there doesn't seem to be other way.

I could probably write the plugin myself that would do the workaround and create the injectors, but I do think this is more of a hack and shouldn't be the accepted solution.

I might later go around and write the injector plugin, it could be something as:

something like new WorkerInjectorGeneratorPlugin({name: "worker.[hash].injector.js", importScripts: ["urlToLoad.[hash].js", secondURLToLoad.[hash].js"])

refer to this issues for reference, and why it should be fixed within webpack and something as a WorkerInjectorGeneratorPlugin would be pretty much a hack plugin.

https://github.com/webpack/webpack/issues/6472

Answer 2

This is really bad answer, but i've managed to share chunks between workers and main thread.

The clue is that

1. globalObject has to be defined as above to (self || this):

output: {
    globalObject: "(self || this)"
}

2. Webpack loads chunks with document.createElement('script') and document.head.appendChild() sequence, which is not available in worker context, but we have self.importScript. So it's just a matter of "polyfiling" it. Here is working "polyfill" (straight from the hell):

console.log("#faking document.createElement()");
(self as any).document = {
    createElement(elementName: string): any {
        console.log("#fake document.createElement", elementName);
        return {};
    },
    head: {
        appendChild(element: any) {
            console.log("#fake document.head.appendChild", element);
            try {
                console.log("loading", element.src);
                importScripts(element.src);
                element.onload({
                    target: element,
                    type: 'load'
                })
            } catch(error) {
                element.onerror({
                    target: element,
                    type: 'error'
                })
            }
        }
    }
};

3. Ensure, that your real code is resolved after polyfill is installed, by using dynamic import, which will. Assuming, that normal "worker main" is in "./RealWorkerMain", that would be "main worker script":

// so, typescript recognizes this as module
export let dummy = 2;

// insert "polyfill from hell" from here

import("./RealWorkerMain").then(({ init }) => {
    init();
});

4. You may need to configure dynamic import in webpack, as documented here is not easy too, this answer was very helpful.

Answer 3

You're looking for universal library target, aka umd.

This exposes your library under all the module definitions, allowing it to work with CommonJS, AMD and as global variable.

To make your Webpack bundle compile to umd you should configure output property like this:

output: {
    filename: '[name].bundle.js',
    libraryTarget: 'umd',
    library: 'yourName',
    umdNamedDefine: true,
},

There is an issue with Webpack 4, but if you still want to use it, you can workaround the issue by adding globalObject: 'this' to the configuration:

output: {
    filename: '[name].bundle.js',
    libraryTarget: 'umd',
    library: 'yourName',
    umdNamedDefine: true,
    globalObject: 'this'
},