René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Proposal: new `vm` module primitives \u0026 loader API for ESM customization","articleBody":"This proposes new vm module primitives that aim to replace the existing `vm.SourceTextModule` and provide a high-level loader API for ESM (specifically `SourceTextModule`) loading customization. \n\nConsider this a very early draft for discussion. This is mostly to investigate whether a new design can addresses the existing issues. I am not 100% it's implementable yet, especially the loader customization part, but it's better to discuss what design would help developers before we think about what is easier to implement.\n\nThere'll be a session about this new design at the collaboration summit too https://github.com/openjs-foundation/summit/issues/482\n\n## Background\n\nThe `vm` module APIs have been behind `--experimental-vm-modules` for a long time. There was a [tracking issue](https://github.com/nodejs/node/issues/37648) about their stabilization and accumulated several issues over the years.\n\nThe meaningful current users are concentrated in test tooling (Jest, Vitest). Recently there has been renewed momentum to look into what it takes to bring the API out of the experimental status. @legendecas and I have been adding some non-breaking changes e.g. `linkRequests()`, `instantiate()`, `moduleRequests`, `hasTopLevelAwait()`, `hasAsyncGraph()`, conditionally synchronous `evaluate()` so that it now provides capabilities required to implement something similar to how ESM is handled in the built-in loader - specifically, the linking process can be driven by those who construct the `SourceTextModule` instead of being driven from a `link` method with callbacks, and it can be conditionally synchronous as the spec allows. But it has become awkward to keep piling methods on the existing classes to do things differently without breaking the API.\n\nThere are still a few issues with the current design:\n\n1. The `importModuleDynamically` and `initializeImportMeta` callbacks are passed as options to the module constructors, requiring careful memory management to avoid leaks when callbacks capture over referrers ([#33439](https://github.com/nodejs/node/issues/33439), [#50113](https://github.com/nodejs/node/issues/50113), [#59118](https://github.com/nodejs/node/issues/59118)), or use-after-free when remote code calls `import()` indirectly via a closure ([#47096](https://github.com/nodejs/node/issues/47096)). We've addressed for the main context with a [very intricate memory management scheme](https://joyeecheung.github.io/blog/2023/12/30/fixing-nodejs-vm-apis-1/), but for new contexts it's uncertain. The current implementation works but is not very GC-efficient, and this issue may not have existed in the first place if the callbacks are managed differently.\n2. `evaluate()` mixes loader errors (status errors, timeout) with module evaluation errors in a single promise rejection, making it hard to handle them differently ([#60242](https://github.com/nodejs/node/issues/60242)).\n3. Many users have expressed that the current API still requires too much plumbing if they only need partial customization. ([#31234](https://github.com/nodejs/node/issues/31234), [#35848](https://github.com/nodejs/node/issues/35848), [#43899](https://github.com/nodejs/node/issues/43899)).\n\nIt seems better to consolidate the changes into a new API rather than continuing to pile onto the existing interface, while we can still steer the design during the experimental phase.\n\n## A draft for a new API\n\nThe new API can live in a `'vm/modules'` module and exports `SourceTextModule`, `SyntheticModule`, and `SourceTextModuleLoader`.\n\n### Core idea\n\nProvide a `SourceTextModuleLoader` abstraction that users can subclass and override high-level processes they want to customize ([#43899](https://github.com/nodejs/node/issues/43899)). This loader can be used standalone, or registered in a given context:\n\n- `dynamicImport(request, context, parent)`\n- `importMeta(meta, context, parent)`\n- `getModules(requests, context, parent)` (resolving and linking a batch of modules for a set of import requests)\n\nIn this model, `SourceTextModule` and `SyntheticModule` are primitives of the loader.\n\nWhen registered for a given context, the loader is responsible for resolving ESM requests, handling dynamic `import()`, and initializing `import.meta`. Instead of one callback per module, there is one loader per context.\n\nNote that this means once there's a loader registered, the internals have to wrap several constructs to be a publically accessible shape e.g. wrapping actual context into vm Context. So it can take a bit of refactoring and adds overhead, but should be managable.\n\nThis is separate from `module.registerHooks()` which installs low-level hooks into built-in resolution/loading process for all types of modules - consider the hooks run underneath `super.getModules()`/`super.dynamicImport()` etc. as shown below, so they operate in a different layer. `SourceTextModuleLoader` is a higher level customization, and as the name implies, is specifically for handling `SourceTextModule`s - `SyntheticModule` does not have `import()` or `import.meta` or load other modules, so they are not applicable to the customization.\n\n### 1. Full customization\n\n```js\nimport {\n  SourceTextModule,\n  SyntheticModule,\n  SourceTextModuleLoader,\n} from 'vm/modules';\n\n// Names starting with `helper` are only for this example,\n// they are not part of the API.\nclass ExampleModuleLoader extends SourceTextModuleLoader {\n  #sourceStore = new Map();\n  #moduleCache = new Map();\n\n  // Invoked by Node.js to perform dynamic import(), overrides default.\n  dynamicImport(request, context, parent) {\n    const mod = this.helperGetSingleModule(request, context, parent);\n    mod.evaluate();\n    // topLevelCapability does not need to fulfill here.\n    // That promise will just be forwarded to the code actually\n    // awaiting the dynamic import().\n    return mod.namespace();\n  }\n\n  // Invoked by Node.js to initialize import.meta, overrides the default.\n  importMeta(meta, context, parent) {\n    // Objects attached to import.meta should typically be created\n    // in the target context via vm.runInContext() or similar.\n    meta.identifier = parent.identifier;\n  }\n\n  // Invoked by Node.js to resolve and compile modules for a set of\n  // import requests. The returned array must correspond 1:1 to the\n  // requests array.\n  // TODO: it's unclear whether getModules or getModule would work better\n  // for the internal way of resolving modules - supposedly batching helps\n  // with performance, but it also leaks internal ordering choices.\n  getModules(requests, context, parent) {\n    const result = [];\n    for (const { specifier } of requests) {\n      // Consult the custom cache first.\n      const cached = this.#moduleCache.get(specifier);\n      if (cached) { result.push(cached); continue; }\n\n      // Use the custom source store.\n      const source = this.#sourceStore.get(specifier);\n      if (!source) {\n        throw new Error('module not found');\n      }\n\n      const identifier = specifier;\n      const mod = new SourceTextModule(source, { identifier, context });\n\n      // Recursively get dependencies (DFS; users can implement BFS too).\n      const modules = this.getModules(mod.requests, context, mod);\n      mod.link(modules);\n\n      this.#moduleCache.set(specifier, mod);\n      result.push(mod);\n    }\n    return result;\n  }\n\n  // --- Example helpers (not part of the API) ---\n\n  helperGetSingleModule(request, context, parent) {\n    return this.getModules([request], context, parent)[0];\n  }\n\n  helperAddSource(identifier, source) {\n    this.#sourceStore.set(identifier, source);\n  }\n\n  helperAddBuiltin(identifier, mapping) {\n    const keys = Object.keys(mapping);\n    const mod = new SyntheticModule(keys, function() {\n      for (const key of keys) {\n        this.setExport(key, mapping[key]);\n      }\n    });\n    this.#moduleCache.set(identifier, mod);\n  }\n}\n```\n\nPreparing the custom loader:\n\n```js\nconst loader = new ExampleModuleLoader();\n\n// If the loader needs to fetch the source asynchronously,\n// they can pre-fetch the whole graph from a known root,\n// so later when `getModules()` is run, it just loads from the\n// asynchronously prefetched graph synchronously  and sets up\n// the links. This is similar to what Node.js internally does.\nloader.helperAddSource('async-root', `\n  export { foo } from 'foo';\n  export let bar;\n  bar = await import('builtin:bar') + import.meta.identifier;\n`);\n\nloader.helperAddSource('sync-root', `\n  export { foo } from 'foo';\n  import { default as bar } from 'builtin:bar';\n  export const baz = bar + import.meta.identifier;\n`);\n\nloader.helperAddSource('foo', `\n  export const foo = globalThis.foo;\n`);\n\nloader.helperAddBuiltin('builtin:bar', { default: 'bar' });\n```\n\n#### 1.a. Async module graph in a new context\n\n```js\nimport { createContext } from 'node:vm';\nconst context = createContext({ foo: 'foo' });\n\nconst [mod] = loader.getModules([{ specifier: 'async-root' }], context);\n\n// returns undefined or throws for any loader errors.\nmod.evaluate();\n\nmod.hasTopLevelAwait();  // true\nmod.hasAsyncGraph();  // true\n\nmod.namespace();  // { foo: .., bar: ... } - not yet populated\n\n// Module completion is tracked separately via topLevelCapability,\n// so it's what you await to finish evaluation of the module.\nawait mod.topLevelCapability;\nmod.namespace();  // { foo: 'foo', bar: 'barasync-root' }\n```\n\n#### 1.b. Synchronous require(esm) pattern\n\nSee [#59656](https://github.com/nodejs/node/issues/59656) for this use case.\n\n```js\nconst [mod] = loader.getModules([{ specifier: 'sync-root' }], context);\n\nmod.evaluate();  // returns undefined or throws for any loader errors.\n\nmod.hasTopLevelAwait();  // false\nmod.hasAsyncGraph();  // false\nmod.topLevelCapability;  // Always fulfilled for non-TLA modules.\nmod.error;  // If the module threw during evaluation, the error is here.\nmod.namespace();  // { foo: 'foo', baz: 'barsync-root' }\n```\n\n### 2. Partial customization and registering globally\n\nA loader can delegate to the base class for specifiers it doesn't need to customize, and can be registered for a context so that all **ESM** running in that context (including `vm.Script` dynamic `import()`s) uses it.\n\n```js\nimport { SourceTextModuleLoader, SyntheticModule } from 'vm/modules';\nimport { createContext, runInContext } from 'node:vm';\n\nclass CustomLoader extends SourceTextModuleLoader {\n  getModules(requests, context, parent) {\n    return requests.map((req) =\u003e {\n      if (req.attributes?.type === 'foo') {\n        return new SyntheticModule(['foo'], function() {\n          this.setExport('foo', 'foo');\n        }, { context });\n      }\n      return super.getModules([req], context, parent)[0];\n    });\n  }\n  // Other methods are left to the default.\n}\n\nconst loader = new CustomLoader();\n\n// Register the loader for a vm context.\n// Only one loader can be registered per context.\nloader.register(context);\n\nconsole.log(\n  await runInContext(\n    `import('foo', { with: { type: 'foo' } })`,\n    context\n  )\n);  // { foo: 'foo' }\n\n// Unregister when done.\nloader.deregister(context);\n\n// Register for the main context.\nloader.register();\nconsole.log(await import('foo', { with: { type: 'foo' } }));\n```\n\n## Design details\n\n### Error model: loader errors vs. module errors\n\nSee [#60242](https://github.com/nodejs/node/issues/60242) for background.\n\nIn the current API, `evaluate()` wraps all errors (status errors, timeout, and actual module exceptions) into a single promise rejection. This makes it difficult to distinguish errors from the loader (implemented by e.g. a framework) from module-level errors (thrown from e.g. user-provided code being tested by a framework), and makes re-implementation of `require(esm)` awkward.\n\nIn the new API, `evaluate()` separates the two:\n\n- Synchronous throws for loader/infrastructure errors: wrong module status, timeout (`ERR_SCRIPT_EXECUTION_TIMEOUT`), signal interruption (`ERR_SCRIPT_EXECUTION_INTERRUPTED`).\n- Use `module.topLevelCapability` (maps to `CyclicModuleRecord.[[TopLevelCapability]]` in the spec) to access evaluation resolution or rejections\n- Use `module.error` to access `CyclicModuleRecord.[[EvaluationError]]` once it's evaluated.\n\n### Source phase imports\n\nModule requests carry phase information (`request.phase`). For source-phase static imports (`import source x from 'y'`), the module returned from `getModules()` must have a source object set via `mod.setSourceObject(obj)`. For dynamic source-phase imports (`import.source('y')`), `dynamicImport()` receives `request.phase === 'source'` and should return the source object (e.g., a `WebAssembly.Module`) instead of a namespace.\n\n```js\ndynamicImport(request, context, parent) {\n  const mod = this.helperGetSingleModule(request, context, parent);\n  if (request.phase === 'source') {\n    return mod.sourceObject;\n  }\n  mod.evaluate();\n  return mod.namespace();\n}\n```\n\nTODO: figure out what to do for deferred imports, but it'll be a phase as well.\n\n### One loader per context\n\nWhen a loader is registered for a context, it handles dynamic `import()` for all code evaluated in that context, including `vm.Script`. With the new design, omitting `importModuleDynamically` from a `vm.Script` should mean \"delegate to the context's registered loader if any, otherwise throw.\" This makes `loader.register(context)` a single point of configuration for all ESM loading in a given context.\n\nThe new `dynamicImport(request, context, parent)` callback receives the context as a parameter, so it can construct modules in the right context. This addresses the issue where `importModuleDynamically` for `vm.Script` didn't have context information ([#35714](https://github.com/nodejs/node/issues/35714)).\n\n`loader.register(context)` throws if a loader is already registered for that context. For users who want hooks-style middleware composition, `module.registerHooks(hooks, context)` is still a a separate, orthogonal mechanism that allow nesting, and it runs in a lower layer underneath the loader's `getModules()`/`dynamicImport()`.\n\n### Different level of customizations\n\nSee [#31234](https://github.com/nodejs/node/issues/31234), [#35848](https://github.com/nodejs/node/issues/35848), [#43899](https://github.com/nodejs/node/issues/43899) [#61127](https://github.com/nodejs/node/issues/61127).\n\nThe base `SourceTextModuleLoader` needs a meaningful default `getModules()` implementation for partial customization to work. The plan is to expose Node's built-in resolution and loading as composable functions (someting like `module.resolve(specifier, parentURL, context)` and `module.load(url, context)`) that the base `getModules()` use (see https://github.com/nodejs/node/pull/55756). This way:\n\n- Partial customizers call `super.getModules()` for requests they don't handle.\n- Advanced users can call `module.resolve` / `module.load` directly to implement their `getModules()` override, these in turn runs the hooks registered by `module.registerHooks()` and/or the built-in resolution/loading logic, so the hooks and the loader are composable in a flexible way.\n\n","author":{"url":"https://github.com/joyeecheung","@type":"Person","name":"joyeecheung"},"datePublished":"2026-04-13T16:48:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":16},"url":"https://github.com/62720/node/issues/62720"}