How can I manually compile a svelte component down to the final javascript and css that sapper/svelte produces?

Question

Our company produces an automation framework that is written in svelte/sapper. One feature is that developers can create custom ui widgets, currently using plain js/html/css and our client side api. These widgets are stored in the database and not on the file system.

I think it would be a big plus to allow them to create widgets as svelte components since it contains all of the markup, js and css in one location and would give them all of the benefits of svelte's reactivity.

I have gotten as far as creating an endpoint that compiles components using svelte's server API but that just seems to generate a module that is ready for rollup-plugin-svelte/sapper/babel to finish the job of producing something the browser can use.

How can I manually compile a svelte component down to the final javascript and css that sapper/svelte produces.

Answer 1

Ouch, tough one. Hang tight.

What you're missing actually is the "linking", that is resolving import statements in the compiled code to something the browser can use. This is the work that is typically done by the bundler (e.g. Rollup, Webpack...).

These imports can come from user (widget developer) code. For example:

import { onMount } from 'svelte'
import { readable } from 'svelte/store'
import { fade } from 'svelte/transition'
import Foo from './Foo.svelte'

Or they can be injected by the compiler, depending on the features that are used in your component. For example:

// those ones are inescapable (bellow is just an example, you'll 
// get different imports depending on what the compiled component 
// actually does / uses)
import {
  SvelteComponent,
  detach,
  element,
  init,
  insert,
  noop,
  safe_not_equal,
} from 'svelte/internal'

Svelte compiles .svelte to .js and, optionally, .css, but it doesn't do anything with imports in your code. On the contrary, it adds some (but still, doesn't resolve them, it's out of its scope).

You'd need to parse the compiled code to find those imports that, raw from the compiler, probably points to paths on your file system and your node_modules directory, and rewrite them to something that makes sense for the browser -- that is, URLs...

Doesn't seem much fun, does it? (Or too much of it, depending on how you see things...) Fortunately, you're not alone with this need and we've got pretty powerful tooling dedicated precisely to this task: enter the bundler!

Solving the linking problem

One relatively straightforward approach to this problem (more to come, don't get too excited too early) is to compile your widgets, not with Svelte's compiler API, but with Rollup and the Svelte plugin.

The Svelte plugin essentially does what you were doing with the compiler API, but Rollup will also do all the hard work of rewiring imports and dependencies in order to produce a neat little package (bundle) that is consumable by the browser (i.e. that doesn't rely on your file system).

You can compile one widget (here Foo.svelte) using some Rollup config like this:

rollup.config.Foo.js

import svelte from 'rollup-plugin-svelte'
import commonjs from '@rollup/plugin-commonjs'
import resolve from '@rollup/plugin-node-resolve'
import css from 'rollup-plugin-css-only'
import { terser } from 'rollup-plugin-terser'

const production = !process.env.ROLLUP_WATCH

// include CSS in component's JS for ease of use
//
// set to true to get separate CSS for the component (but then,
// you'll need to inject it yourself at runtime somehow)
//
const emitCss = false

const cmp = 'Foo'

export default {
  // our widget as input
  input: `widgets/${cmp}.svelte`,

  output: {
    format: 'es',
    file: `public/build/widgets/${cmp}.js`,
    sourcemap: true,
  },

  // usual plugins for Svelte... customize as needed
  plugins: [
    svelte({
      emitCss,
      compilerOptions: {
        dev: !production,
      },
    }),

    emitCss && css({ output: `${cmp}.css` }),

    resolve({
      browser: true,
      dedupe: ['svelte'],
    }),
    commonjs(),
    production && terser(),
  ],
}

Nothing very extraordinary here... This is basically the config from the official Svelte template for Rollup, minus the parts pertaining to the dev server.

Use the above config with a command like this:

rollup --config rollup.config.Foo.js

And you'll get your browser-ready compiled Foo widget in public/build/Foo.js!

Rollup also has a JS API so you could run this programmatically as needed from a web server or whatever.

Then you'll be able to dynamically import and then use this module with something like this in your app:

const widget = 'Foo'
const url = `/build/widgets/${widget}.js`

const { default: WidgetComponent } = await import(url)

const cmp = new WidgetComponent({ target, props })

Dynamic imports will probably be necessary in your case, because you won't know about the widgets at the time you build your main app -- hence you will need to construct the import URLs dynamically like above at runtime. Note that the fact that the import URL is a dynamic string will prevent Rollup from trying to resolve it at bundle time. This means the import will end up as written above in the browser, and that it must be an URL (not a file path on your machine) that the browser will be able to resolve.

That's because we're consuming the compiled widget with a browser native dynamic import that we need to set output.format to es in the Rollup config. The Svelte component will be exposed with export default ... syntax, that modern browsers natively understand.

Dynamic imports are very well supported by current browsers. The notable exception is the "old" Edge (before it essentially became Chrome). If you need to support older browsers, polyfills are available (many of them actually -- e.g. dimport).

This config can be further automatized to be able to compile any widget, not just Foo. For example, like this:

rollup.config.widget.js

... // same as above essentially

// using Rollup's --configXxx feature to dynamically generate config
export default ({ configWidget: cmp }) => ({
  input: `widgets/${cmp}.svelte`,
  output: {
    ...
    file: `public/build/widgets/${cmp}.js`,
  },
  ...
})

You can then use it like this:

rollup --config rollup.config.widget.js --configTarget Bar

We're making progress, yet there remains a few caveats and hurdles to be aware of (and maybe optimize further -- your call).

Caveat: shared dependencies

The above approach should give you the compiled code for your widgets, that you can run in the browser, with no unresolved imports. Good. However, it does so by resolving all dependencies of a given widget when it is built, and bundling all these dependencies in the same file.

Said otherwise, all dependencies that are shared between multiple widgets will be duplicated for every widget, very notably the Svelte dependencies (i.e. imports from svelte or svelte/*). This is not all bad, because it gives you very standalone widgets... Unfortunately, this also add some weight to your widgets code. We're talking something like maybe 20-30 kb of JS added to each widgets that could be shared between all of them.

Also, as we will see soon, having independent copies of Svelte internals in your app has some drawbacks we need to take into consideration...

One easy way to extract common dependencies so they can be shared instead of duplicated is to bundle all your widgets in one pass. This might not be practicable for all the widgets of all your users, but maybe it can be doable at individual user level?

Anyway, here's the general idea. You would change the above Rollup configs to something like this:

rollup.config.widget-all.js

...

export default {
  input: ['widgets/Foo.svelte', 'widgets/Bar.svelte', ...],
  output: {
    format: 'es',
    dir: 'public/build/widgets',
  },
  ...
}

We're passing an array of files, instead of just one, as the input (you would probably automatize this step by listing files in a given directory), and we're changing output.file to output.dir, since now we're gonna have several files generated at once. Those files will include common dependencies of your widgets that Rollup will have extracted, and that all your widgets will share between them for reuse.

Further perspectives

It would be possible to push even further, by extracting some shared dependencies (say, Svelte...) yourself and make them available as URLs to the browser (i.e. serve them with your web server). This way, you could rewrite those imports in your compiled code to those known URLs instead of relying on Rollup to resolve them.

This would reduce code duplication entirely, saving weight, and also this would allow to have a single version of those dependencies shared among all the widget that use them. Doing so would also relieve the need to build all widgets that share dependencies in one go at the same time, which is alluring... However, this would be pretty (!) complicated to setup, and you would actually hit diminishing returns fast.

In effect, when you're bundling a bunch of widgets together (or even just one) and let Rollup extract the dependencies, it is possible for the bundler to know what parts of the dependencies are actually needed by the consuming code and skip the rest (keep in mind: Rollup was built with tree shaking as one -- if not the one -- of its main priority, and Svelte was built by the same guy -- meaning: you can expect Svelte to be very tree shaking friendly!). On the other hand, if you extract some dependencies manually yourself: it relieves the need to bundle all consuming code at once, but you will have to expose the whole of the consumed dependencies, because you won't be able to know in advance the parts from them that will be needed.

It's a balance you need to find between what is efficient and what is practical, accounting for the added complexity of each solution to your setup. Given your use case, my own feeling is that the sweet spot is either bundling each widget entirely independently, or bundling a bunch of widgets from, say, the same user together to save some weight, as described above. Pushing harder would probably be an interesting technical challenge, but it would reap just little extra benefits, but somewhat exploding complexity...

OK so we now know how to bundle our widgets for the browser. We even have some degree of control on how to pack our widgets entirely standalone, or take on some extra infrastructure complexity to rather share dependencies between them and save some weight. Now, we've got a special dependency to consider, when we decide how we make our pretty little packets (err, bundles): that's Svelte itself...

Mind the trap: Svelte can't be duplicated

So we understand that when we're bundling a single widget with Rollup, all of its dependencies will be included in the "bundle" (just the one widget file in this case). If you bundle 2 widgets in this way and they share some dependencies, those dependencies will be duplicated in each of those bundles. In particular, you'd get 2 copies of Svelte, one in each widget. Likewise, the dependencies of your "main" app that are shared with some widgets will nonetheless be duplicated in the browser. You'll have multiple copies of the same code that will be used by those different bundles -- your app, different widgets...

However, there is something special about Svelte that you need to know: it doesn't support being duplicated. The svelte/internal module is stateful, it contains some global variables that would be duplicated if you have multiple copies of this code (see above). What this means, in practice, is that Svelte components that don't use the same copie of Svelte internals can't be used together.

For example, if you have a App.svelte component (your main app) and a Foo.svelte component (e.g. a user widget) that have been bundled independently, then you can't use Foo in App, or you'd get weird bugs.

This wouldn't work:

App.svelte

<script>
  // as we've seen, in real life, this would surely be a 
  // dynamic import but whatever, you get the idea
  import Foo from '/build/widgets/Foo.js'
</script>

<!-- NO -->
<Foo />

<!-- NO -->
<svelte:component this={Foo} />

That's also the reason why you have this dedupe: ['svelte'] option in the official Svelte template's Rollup config... This is intended to prevent bundling different copies of Svelte, which would happen if you ever used linked packages, for example.

Anyway, in your case it is kind of unescapable to end up with multiple copies of Svelte in the browser, since you're probably not wanting to rebuild your whole main app anytime a user adds or changes one of their widget... Except going to great lengths to extract, centralize, and rewrite the Svelte imports yourself; but, as I said, I don't believe this would be a reasonable and sustainable approach.

And so we're stuck.

Or are we?

The problem of duplicated Svelte copies only occurs when the conflicting components are part of the same components tree. That is, when you let Svelte create and manage the component instances, like above. The problem doesn't exist when you create and manage the component instances yourself.

...

const foo = new Foo({ target: document.querySelector('#foo') })

const bar = new Bar({ target: document.querySelector('#bar') })

Here foo and bar will be entirely independent component trees, as far as Svelte is concerned. Code like this will always work, irrelevantly of how and when (and with which Svelte version, etc.) Foo and Bar were compiled and bundled.

As I understand your use case, this is not a major hurdle. You won't be able to embed your users' widgets into your main app with something like <svelte:component />... However, nothing prevents you from creating and managing the widget instances in the right place yourself. You can create a wrapper component (in your main app) to generalize this approach. Something like this:

Widget.svelte

<script>
  import { onDestroy } from 'svelte'

  let component
  export { component as this }

  let target
  let cmp

  const create = () => {
    cmp = new component({
      target,
      props: $$restProps,
    })
  }

  const cleanup = () => {
    if (!cmp) return
    cmp.$destroy()
    cmp = null
  }

  $: if (component && target) {
    cleanup()
    create()
  }

  $: if (cmp) {
    cmp.$set($$restProps)
  }

  onDestroy(cleanup)
</script>

<div bind:this={target} />

We create a target DOM element from our main app, render an "external" component in it, pass down all the props (we're proxying reactivity), and don't forget to cleanup when our proxy component is destroyed.

The main limitation of such an approach is that Svelte context (setContext / getContext) of the app won't be visible to the proxied components.

Once again, this doesn't really seem like a problem in the widget use case -- maybe even better: do we really want the widgets to have access to every bits of the surrounding app? If really needed, you can always pass bits of context down to the widget components via props.

The above Widget proxy component would then be used like this in your main app:

<script>
  import Widget from './Widget.svelte'

  const widgetName = 'Foo'

  let widget

  import(`/build/widgets/${widgetName}.js`)
    .then(module => {
      widget = module.default
    })
    .catch(err => {
      console.error(`Failed to load ${widgetName}`, err)
    })
</script>

{#if widget}
  <Widget this={widget} prop="Foo" otherProp="Bar" />
{/if}

And... Here we are? Let's sum it up!

Summary

• Compile your widgets with Rollup, not Svelte compiler directly, to produce browser ready bundles.

• Find the right balance between simplicity, duplication and extra weight.

• Use dynamic imports to consume your widgets, that will be built independently of your main app, in the browser.

• Do not try to mix together components that don't use the same copy of Svelte (essentially means bundled together, except if you've launched into some extraordinary hack). It might looks like it works at first, but it won't.