René's URL Explorer Experiment

Title: fs: read small files in one thread pool round trip by codebytere · Pull Request #65327 · nodejs/node · GitHub

Open Graph Title: fs: read small files in one thread pool round trip by codebytere · Pull Request #65327 · nodejs/node

X Title: fs: read small files in one thread pool round trip by codebytere · Pull Request #65327 · nodejs/node

Description: fs.readFile() / fs.promises.readFile() of small files get 3–6× faster, and use one libuv thread-pool task instead of four, by doing open + fstat + read + close in a single round trip. fs/readfile.js len=1024 concurrent=1 *** 204.95 % ±2.29% fs/readfile.js len=1024 concurrent=10 *** 299.99 % ±4.58% fs/readfile-promises.js len=1024 concurrent=1 *** 238.56 % ±2.68% fs/readfile-promises.js len=1024 concurrent=10 *** 500.14 % ±5.17% fs/readfile-promises.js len=524288 concurrent=10 encoding='utf-8' *** 132.83 % ±8.34% fs/readfile-partitioned.js len=1024 concurrent=10 (vs. zlib work) *** 240.87 % fs/readfile*.js len ≥ 4 MiB ~0 % n.s. (one exception below) fs.readFile() of 4 KiB files at concurrency 64: ~51 k → ~306 k files/s; mixed with fs.stat + dns.lookup: ~66 k → ~312 k ops/s (Linux x64, --set duration=2, 30 runs.) Today a path-based readFile issues open, fstat, read and close as four separate uv_fs_* requests, each with its own queue wait, completion callback and JS↔C++ crossing; the promise API does the same through a FileHandle. For small files those round trips are the whole cost, and each one takes a pool slot away from concurrent dns/zlib/crypto/fs work. ReadFileJob (an AsyncWrap + ThreadPoolWork, provider FSREQCALLBACK) runs open + fstat + read-to-EOF + close as one task and returns the content. If the file is larger than one chunk (kReadFileBufferLength, 512 KiB) it stops after fstat and hands back the fd and size, and the existing chunked reader continues exactly as today (interleaved, abortable between chunks). Both readFiles use it for path arguments without a user buffer; fds and FileHandles are unchanged. Preserved on purpose: identical results for every size/encoding; open errors report syscall: 'open' + path, read errors 'read'; permission errors arrive through the callback/promise; an abort that lands while the round trip is in flight still wins; the handed-back fd is tracked and closed like any other; size-0 files (procfs) are read to EOF. One open point: 16–32 MiB reads via fs.promises.readFile(…, 'utf-8') at concurrency 10 measure −2…3 % (***), reproducibly; the same sizes as Buffers, via the callback API, or at concurrency 1 are flat. They take the hand-back path with identical syscalls, and direct timing shows ≤2 %, so I haven't pinned it down. If preferred, the promise API can keep its current path and only the callback API changes. Tests: test-fs-readfile-one-roundtrip.js (new; also passes on current main): sizes across the 512 KiB threshold, encodings, flags, error shapes, abort before/during, fd/FileHandle inputs, no fd leak on hand-back, procfs/sysfs, async_hooks lifecycle. Adjusted to keep testing what they test: test/async-hooks/test-fsreqcallback-readFile.js (accepts one request instead of exactly four), test-graph.fsreq-readFile.js (reads a 512 KiB+1 file so the four-request chain keeps its shape), test-async-exec-resource-match.js (resource + ≥1 fs request), test-trace-events-fs-async.js (uses fs.fstat() for the fstat trace instead of readFile as a proxy), test-fs-promises-readfile.js (zero-size-liar case goes through a FileHandle), test-fs-promises-file-handle-{op,aggregate,close}-errors.js (use a >512 KiB file so the patched FileHandle path is taken). fs, async-hooks, permission, worker, process and child_process suites pass. Disclosure: the code, test, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.

Open Graph Description: fs.readFile() / fs.promises.readFile() of small files get 3–6× faster, and use one libuv thread-pool task instead of four, by doing open + fstat + read + close in a single round trip. fs/readfile.j...

X Description: fs.readFile() / fs.promises.readFile() of small files get 3–6× faster, and use one libuv thread-pool task instead of four, by doing open + fstat + read + close in a single round trip. fs/readfile.j...

Opengraph URL: https://github.com/nodejs/node/pull/65327

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width