René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"fs: caller-supplied buffer for fs.readFile / fs.readFileSync","articleBody":"`fs.readFile*` allocates a fresh Buffer on every call. Fine for most code, expensive on hot paths. Multiple Worker threads doing concurrent reads contend hard on the per-process `mmap_lock` in the kernel — about 17× longer wait per allocation than equivalent child processes (trace in #63597).\n\nThe workaround today is `openSync` + `readSync` + `closeSync` with your own buffer. Works, but you lose `readFile`'s ergonomics, especially for the async variants.\n\nTwo small additions would close that gap.\n\n### `{ buffer }`\n\nPass a buffer to fill, get back a `subarray` view of the actual file bytes:\n\n```js\nimport fs from 'node:fs';\nimport fsp from 'node:fs/promises';\n\nconst buf = Buffer.allocUnsafe(1 \u003c\u003c 20);\n\n// sync\nconst data = fs.readFileSync(path, { buffer: buf });\n// data === buf.subarray(0, fileSize)\n\n// promise\nconst data2 = await fsp.readFile(path, { buffer: buf });\n\n// callback\nfs.readFile(path, { buffer: buf }, (err, data3) =\u003e { /* ... */ });\n```\n\nIf the file is bigger than the buffer, throw. Same shape as `crypto.randomFillSync(buf)` vs `crypto.randomBytes(n)`, or `fs.read(fd, buf, ...)` at the lower level.\n\n### `{ getBuffer(size) }`\n\nPass a factory called with the file's size (we already get it from `fstat`):\n\n```js\nimport fs from 'node:fs';\nimport fsp from 'node:fs/promises';\n\nconst opts = { getBuffer(size) { return pool.acquire(size); } };\n\nconst data  = fs.readFileSync(path, opts);\nconst data2 = await fsp.readFile(path, opts);\nfs.readFile(path, opts, (err, data3) =\u003e { /* ... */ });\n```\n\nFor pool implementations that want to pick the right slab once they know the size, instead of pre-allocating one giant buffer per concurrent caller.\n\n`getBuffer` is synchronous in all variants. The async forms still benefit because the allocation step happens inline before the libuv read kicks off.\n\n### Why both\n\n`{ buffer }` is for the simple case (one reusable buffer per worker or stream). `{ getBuffer }` is for pool implementations. They compose:\n\n```\nbuffer    → use it\ngetBuffer → call it\nneither   → allocate (today's behaviour, unchanged)\n```\n\nPassing both throws.\n\n### Things to figure out\n\n* `encoding` + `buffer`: I'd say return the string like today, treat the buffer as scratch.\n* Pool release: leave to userland for v1. The caller can wrap the returned subarray however they want.\n\nImplementation order would be `readFileSync` first, then `fsPromises.readFile`, then callback-style `fs.readFile`.\n\ncc @nodejs/fs @nodejs/performance","author":{"url":"https://github.com/mcollina","@type":"Person","name":"mcollina"},"datePublished":"2026-05-27T12:23:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/63600/node/issues/63600"}