ES Module Loader Polyfill

Provides low-level hooks for creating ES module loaders, roughly based on the API of the WhatWG loader spec, but with adjustments to match the current proposals for the HTML modules specification, unspecified WhatWG changes, and NodeJS ES module adoption.

Supports the loader import and registry API with the System.register module format to provide exact module loading semantics for ES modules in environments today. In addition, support for the System.registerDynamic is provided to allow the linking of module graphs consisting of inter-dependent ES modules and CommonJS modules with their respective semantics retained.

This project aims to provide a fast, minimal, unopinionated loader API on top of which custom loaders can easily be built.

See the spec differences section for a detailed description of some of the specification decisions made.

ES6 Module Loader Polyfill, the previous version of this project was built to the outdated ES6 loader specification and can still be found at the 0.17 branch.

Module Loader Examples

Some examples of common use case module loaders built with this project are provided below:

Browser ES Module Loader: A demonstration-only loader to load ES modules in the browser including support for the <script type="module"> tag as specified in HTML.
Node ES Module Loader Allows loading ES modules with CommonJS interop in Node via node-esml module/path.js in line with the current Node plans for implementing ES modules. Used to run the tests and benchmarks in this project.
System Register Loader: A fast optimized production loader that only loads System.register modules, recreating ES module semantics with CSP support.

Installation

npm install es-module-loader --save-dev

Creating a Loader

This project exposes a public API of ES modules in the core folder.

The minimal polyfill loader API is provided in core/loader-polyfill.js. On top of this main API file is core/register-loader.js which provides a base loader class with the non-spec System.register and System.registerDynamic support to enable the exact linking semantics.

Helper functions are available in core/resolve.js and core/common.js. Everything that is exported can be considered part of the publicly versioned API of this project.

Any tool can be used to build the loader distribution file from these core modules - Rollup is used to do these builds in the example loaders above, provided by the rollup.config.js file in the example loader repos listed above.

Base Loader Polyfill API

The Loader and ModuleNamespace classes in core/loader-polyfill.js provide the basic spec API method shells for a loader instance loader:

new Loader(): Instantiate a new loader instance. Defaults to environment baseURI detection in NodeJS and browsers.
loader.import(key [, parentKey]): Promise for importing and executing a given module, returning its module instance.
loader.resolve(key [, parentKey]): Promise for resolving the idempotent fully-normalized string key for a module.
new ModuleNamespace(bindings): Creates a new module namespace object instance for the given bindings object. The iterable properties of the bindings object are created as getters returning the corresponding values from the bindings object.
loader.registry.set(resolvedKey, namespace): Set a module namespace into the registry.
loader.registry.get(resolvedKey): Get a module namespace (if any) from the registry.
loader.registry.has(resolvedKey): Boolean indicating whether the given key is present in the registry.
loader.registry.delete(resolvedKey): Removes the given module from the registry (if any), returning true or false.
loader.registry.keys(): Function returning the keys iterator for the registry.
loader.registry.values(): Function returning the values iterator for the registry.
loader.registry.entries(): Function returning the entries iterator for the registry (keys and values).
loader.registry[Symbol.iterator]: In supported environments, provides registry entries iteration.

Example of using the base loader API:

import { Loader, ModuleNamespace } from 'es-module-loader/core/loader-polyfill.js';

let loader = new Loader();

// override the resolve hook
loader[Loader.resolve] = function (key, parent) {
  // intercept the load of "x"
  if (key === 'x') {
    this.registry.set('x', new ModuleNamespace({ some: 'exports' }));
    return key;
  }
  return Loader.prototype[Loader.resolve](key, parent);
};

loader.import('x').then(function (m) {
  console.log(m.some);
});

RegisterLoader Hooks

Instead of just hooking modules within the resolve hook, the RegisterLoader base class provides an instantiate hook to separate execution from resolution and enable spec linking semantics.

Implementing a loader on top of the RegisterLoader base class involves extending that class and providing these resolve and instantiate prototype hook methods:

import RegisterLoader from 'es-module-loader/core/register-loader.js';
import { ModuleNamespace } from 'es-module-loader/core/loader-polyfill.js';

class MyCustomLoader extends RegisterLoader {
  /*
   * Constructor
   * Purely for completeness in this example
   */
  constructor (baseKey) {
    super(baseKey);
  }

  /*
   * Default resolve hook
   *
   * The default parent resolution matches the HTML spec module resolution
   * So super[RegisterLoader.resolve](key, parentKey) will return:
   *  - undefined if "key" is a plain names (eg 'lodash')
   *  - URL resolution if "key" is a relative URL (eg './x' will resolve to parentKey as a URL, or the baseURI)
   *
   * So relativeResolved becomes either a fully normalized URL or a plain name (|| key) in this example
   */
  [RegisterLoader.resolve] (key, parentKey) {
    var relativeResolved = super[RegisterLoader.resolve](key, parentKey, metadata) || key;
    return relativeResolved;
  }

  /*
   * Default instantiate hook
   *
   * This is one form of instantiate which is to return a ModuleNamespace directly
   * This will result in every module supporting:
   *
   *   import { moduleName } from 'my-module-name';
   *   assert(moduleName === 'my-module-name');
   */
  [RegisterLoader.instantiate] (key) {
    return new ModuleNamespace({ moduleName: key });
  }
}

The return value of resolve is the final key that is set in the registry.

The default normalization provided (super[RegisterLoader.resolve] above) follows the same approach as the HTML specification for module resolution, whereby plain module names that are not valid URLs, and not starting with ./, ../ or / return undefined.

So for example lodash will return undefined, while ./x will resolve to [baseURI]/x. In NodeJS a file:/// URL is used for the baseURI.

Instantiate Hook

Using these three types of return values for the RegisterLoader instantiate hook, we can recreate ES module semantics interacting with legacy module formats:

1. Instantiating Dynamic Modules via ModuleNamespace

If the exact module definition is already known, or loaded through another method (like calling out fully to the Node require in the node-es-module-loader), then the direct module namespace value can be returned from instantiate:

import { ModuleNamespace } from 'es-module-loader/core/loader-polyfill.js';

// ...

  instantiate (key) {
    var module = customModuleLoad(key);

    return new ModuleNamespace({
      default: module,
      customExport: 'value'
    });
  }

2. Instantiating ES Modules via System.register

When instantiate returns undefined, it is assumed that the module key has already been registered through a loader.register(key, deps, declare) call, following the System.register module format.

For example:

  [RegisterLoader.instantate] (key) {
    // System.register
    this.register(key, ['./dep'], function (_export) {
      // ...
    });
  }

When using the anonymous form of System.register - loader.register(deps, declare), in order to know the context in which it was called, it is necessary to call the processAnonRegister method passed to instantiate:

  [RegisterLoader.instantiate] (key, processAnonRegister) {
    // System.register
    this.register(deps, declare);

    processAnonRegister();
  }

The loader can then match the anonymous System.register call to correct module in the registry. This is used to support <script> loading.

System.register is not designed to be a handwritten module format, and would usually generated from a Babel or TypeScript conversion into the "system" module format.

3. Instantiating Legacy Modules via System.registerDynamic

This is identical to the System.register process above, only running loader.registerDynamic instead of loader.register:

  [RegisterLoader.instantiate] (key, processAnonRegister) {

    // System.registerDynamic CommonJS wrapper format
    this.registerDynamic(['dep'], true, function (require, exports, module) {
      module.exports = require('dep').y;
    });

    processAnonRegister();
  }

For more information on the System.registerDynamic format see the format explanation.

Performance

Some simple benchmarks loading System.register modules are provided in the bench folder:

Each test operation includes a new loader class instantiation, System.register declarations, binding setup for ES module trees, loading and execution.

Sample results:

Test	ES Module Loader 1.3
Importing multiple trees at the same time	654 ops/sec
Importing a deep tree of modules	4,162 ops/sec
Importing a single module with deps	8,817 ops/sec
Importing a single module without deps	16,536 ops/sec

Tracing API

When loader.trace = true is set, loader.loads provides a simple tracing API.

Also not in the spec, this allows useful tooling to build on top of the loader.

loader.loads is keyed by the module ID, with each record of the form:

{
  key, // String, key
  deps, // Array, unnormalized dependencies
  depMap, // Object, mapping unnormalized dependencies to normalized dependencies
  metadata // Object, exactly as from normalize and instantiate hooks
}

Spec Differences

The loader API in core/loader-polyfill.js matches the API of the current WhatWG loader specification as closely as possible, while making a best-effort implementation of the upcoming loader simplification changes as descibred in whatwg/loader#147.

Default normalization and error handling is implemented as in the HTML specification for module loading. Default normalization follows the HTML specification treatment of module keys as URLs, with plain names ignored by default (effectively erroring unless altering this behaviour through the hooks). Errors are cached in the registry, until the delete API method is called for the module that has errored. Resolve and fetch errors throw during the tree instantiation phase, while evaluation errors throw during the evaluation phase, and this is true for cached errors as well in line with the spec - whatwg/html#2595.
A direct ModuleNamespace constructor is provided over the Module mutator proposal in the WhatWG specification. Instead of storing a registry of Module.Status objects, we then store a registry of Module Namespace objects. The reason for this is that asynchronous rejection of registry entries as a source of truth leads to partial inconsistent rejection states (it is possible for the tick between the rejection of one load and its parent to have to deal with an overlapping in-progress tree), so in order to have a predictable load error rejection process, loads are only stored in the registry as fully-linked Namespace objects and not ModuleStatus objects as promises for Namespace objects. The custom private ModuleNamespace constructor is used over the Module.Status proposal to ensure a stable API instead of tracking in-progress specification work.
Linking between module formats does not use zebra striping anymore, but rather relies on linking the whole graph in deterministic order for each module format down the tree as is planned for NodeJS. This is made possible by the dynamic modules TC39 proposal which allows the export named bindings to only be determined at evaluation time for CommonJS modules. We do not currently provide tracking of circular references across module format boundaries so these will hang indefinitely like writing an infinite loop.
Loader is available as a named export from core/loader-polyfill.js but is not by default exported to the global.Reflect object. This is to allow individual loader implementations to determine their own impact on the environment.
A constructor argument is added to the loader that takes the environment baseKey to be used as the default normalization parent.
The RegisterLoader splits up the resolve hook into resolve and instantiate. The WhatWG reduced specification proposal to remove the loader hooks implies having a single resolve hook by having the module set into the registry using the registry.set API as a side-effect of resolution to allow custom interception as in the first loader example above. As discussed in whatwg/loader#147 (comment), this may cause unwanted execution of modules when only resolution is needed via loader.resolve calls, so the approach taken in the RegisterLoader is to implement separate resolve and instantiate hooks.

License

Licensed under the MIT license.

ModuleLoader / es-module-loader