René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"buffer: utilities to check whether a Buffer or string fits in memory before creating it","articleBody":"### What is the problem this feature will solve?\n\nA very common pattern is to accumulate streamed input (an HTTP body, a file, a socket) and then parse it fully in memory: `Buffer.concat(chunks)`, `buf.toString()`, `JSON.parse(text)`. Today the only defenses are static byte limits (`bodyLimit` in Fastify, `limit` in `raw-body`, and so on). There is no supported way to ask \"will this allocation fit in the memory this process actually has?\", so a limit that is fine on a laptop takes a 512 MB container down.\n\nThe building blocks exist (`buffer.constants.MAX_LENGTH`, `buffer.constants.MAX_STRING_LENGTH`, `process.availableMemory()`, `v8.getHeapStatistics().total_available_size`), but the rules for combining them are non-obvious and partly undocumented. I measured the following on Linux with `main` (v27.0.0-pre):\n\n**Buffer allocation has two failure modes, and only one is catchable.**\n\n- When `malloc` returns `NULL` (for example under `ulimit -v`, or on Linux overcommit mode 0 for a single request larger than RAM+swap) Node throws a catchable `RangeError` `ERR_MEMORY_ALLOCATION_FAILED`.\n- Under a cgroup limit the allocation succeeds and the process is `SIGKILL`ed on first write:\n\n  ```\n  $ systemd-run --user --scope -p MemoryMax=512M -p MemorySwapMax=0 node -e '\n    console.log(process.availableMemory() / 2**20 | 0);   // 501\n    const b = Buffer.allocUnsafe(2**30);                   // succeeds\n    b.fill(1);'                                            // Killed, exit 137\n  ```\n\n  `process.availableMemory()` predicted it correctly. Nothing in the docs says to use it for this.\n\n**Decoded strings land in different memory pools depending on the input.**\n\n| `buf.toString('utf8')` input | Where the result lives |\n|---|---|\n| Valid UTF-8, result >= ~1M chars | External (`malloc`), accounted in `external_memory` |\n| Invalid UTF-8, any size | V8 heap (`String::NewFromUtf8` fallback in `src/string_bytes.cc`) |\n| Result \u003c ~1M chars (`EXTERN_APEX`) | V8 heap |\n\nThe same 16 MB body costs native memory when it is well formed and V8 heap when one byte is truncated. `TextDecoder` and `stream/consumers` use the same path.\n\n**The V8 heap limit is soft and its OOM is fatal.** With `--max-old-space-size=128` a decode that needs 267 MB of heap succeeds; the process dies with `SIGABRT` at the next GC that cannot get back under the limit. `JSON.parse` of a 60 MB array of small integers needs ~300 MB of heap and aborts. `v8.getHeapStatistics().total_available_size` (0.5 µs/call) is the only predictor, and it cannot see the object graph a parse will build.\n\n**The `MAX_STRING_LENGTH` check for `latin1`/`ascii` runs after the copy.** `ExternString::NewFromCopy` in `src/string_bytes.cc` mallocs and copies the whole input, then `String::NewExternalOneByte` rejects it. Failing on a 512 MB input takes ~400 ms; a successful 100 MB decode takes ~46 ms. The UTF-8 paths check before doing work.\n\n**OS differences (libuv).**\n\n- Linux: `availableMemory()` is cgroup v1/v2 aware and subtracts current usage (~30 µs/call).\n- macOS: `availableMemory()` is free + inactive + purgeable pages; `constrainedMemory()` only reflects `RLIMIT_AS`/`RLIMIT_DATA`, which `malloc` largely ignores.\n- Windows: `constrainedMemory()` is hard-coded to 0 (Job Object limits are invisible); `availableMemory()` is `ullAvailPhys` and ignores the commit limit. Allocation past the commit limit does fail synchronously, so at least it is catchable there.\n\nV8 offers nothing to smooth this over: `GetHeapStatistics`, `NearHeapLimitCallback`, `SetOOMErrorHandler` and `MemoryPressureNotification` cannot reserve memory or answer \"will N bytes fit\".\n\nRelated: #64646 (`heap_size_limit` larger than the cgroup limit) is the same accounting split seen from the other side.\n\n### What is the feature you are proposing to solve the problem?\n\n1. **`buffer.stringLength(buf[, encoding])`**, the mirror of `Buffer.byteLength()`. Returns the number of UTF-16 code units a decode would produce, without decoding (simdutf `utf16_length_from_utf8`, ~10 GB/s; trivial for `latin1`/`ascii`/`hex`/`base64`). This lets a parser check against `MAX_STRING_LENGTH` and compute the `2 * length` byte bound before doing any work. Whether the result can be one-byte is not needed for an upper bound.\n\n2. **Check `MAX_STRING_LENGTH` before copying in the `latin1`/`ascii` decode path** in `src/string_bytes.cc`. Independent, small fix.\n\n3. **Document a memory-budgeting recipe** in `doc/api/buffer.md` and `doc/api/process.md` rather than adding a boolean `canAllocate()`. A boolean would have to hide the headroom policy and the heap/native split, and would be wrong on macOS and Windows where the probes are weaker. The recipe:\n\n   ```js\n   const { constants: { MAX_LENGTH, MAX_STRING_LENGTH }, stringLength } = require('node:buffer');\n   const RESERVE = 64 * 2 ** 20;  // headroom for the rest of the process\n\n   // (a) Buffer.alloc(n), (d) Buffer.concat(list) with n = total length\n   //     (peak is 2 * n while the sources are alive)\n   function fitsNative(n) {\n     return n \u003c= MAX_LENGTH && n + RESERVE \u003c= process.availableMemory();\n   }\n\n   // (b) a string of `units` UTF-16 code units built in JS (concat, JSON.stringify, ...)\n   function fitsHeap(units) {\n     return units \u003c= MAX_STRING_LENGTH &&\n       units * 2 + RESERVE \u003c= v8.getHeapStatistics().total_available_size;\n   }\n\n   // (c) buf.toString(): large valid UTF-8 is native, everything else is heap\n   function canDecode(buf) {\n     const units = stringLength(buf, 'utf8');\n     return units \u003c= MAX_STRING_LENGTH && fitsNative(units * 2) && fitsHeap(units);\n   }\n   ```\n\n   Plus an explicit warning that nothing can bound `JSON.parse`, and that the V8 limit is enforced at GC time, not at allocation.\n\n4. **libuv follow-up**: read Job Object memory limits in `uv_get_constrained_memory()` on Windows, which is the one gap that makes the recipe wrong in Windows containers.\n\n### What alternatives have you considered?\n\n- `buffer.canAllocate(bytes)` / `Buffer.canAlloc()`: rejected for the reasons in (3). It cannot be made truthful for heap strings and hides OS-specific weakness.\n- A \"reserve\" API that pre-touches pages: would defeat lazy commit and cost a full write of the buffer.\n- Leaving it to userland: userland already does static limits; the information needed for dynamic ones is only partly exposed (`stringLength` does not exist) and the pool split is not documented anywhere.\n\nA portable probe script that reproduces each measurement above (runs the risky steps in child processes) is available on request; I will attach results from macOS and Windows.\n","author":{"url":"https://github.com/mcollina","@type":"Person","name":"mcollina"},"datePublished":"2026-09-16T12:45:43Z","url":"https://github.com/nodejs/node/issues/66062"}