René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Problems with cooperation between http and pipeline/\"for await\"","articleBody":"**Is your feature request related to a problem? Please describe.**\r\n\r\nProblem is that \"for await\" and pipeline destroy source stream on error (\"for await\" also on break) but destroying http request removes possibility to send meaningful response to client.\r\n\r\nExample with \"for await\" (body length limiting):\r\n``` js\r\nlet http = require(\"http\");\r\n\r\nlet server = http.createServer(async function(req, res)\r\n{\r\n\ttry\r\n\t{\r\n\t\tlet len = 0;\r\n\t\tfor await (let chunk of req)\r\n\t\t{\r\n\t\t\tlen += chunk.length;\r\n\t\t\tif(len > 2) throw \"payload too large\";\r\n\t\t}\r\n\t\tres.end(\"ok\");\r\n\t}\r\n\tcatch(err)\r\n\t{\r\n\t\tconsole.log(\"server log:\", err);\r\n\t\tres.end(err);\r\n\t}\r\n});\r\n\r\n(async function()\r\n{\r\n\tawait new Promise(resolve => server.listen(8888, resolve));\r\n\tlet req = http.request({ port: 8888, method: \"post\" });\r\n\treq.end(\"abc\");\r\n\ttry\r\n\t{\r\n\t\tlet res = await new Promise((resolve, reject) => req.on(\"response\", resolve).on(\"error\", reject));\r\n\t\tlet str = \"\";\r\n\t\tfor await (let chunk of res) str += chunk;\r\n\t\tconsole.log(\"client received:\", str);\r\n\t}\r\n\tcatch(err)\r\n\t{\r\n\t\tconsole.log(\"client got unexpected error:\", err);\r\n\t}\r\n\tserver.close();\r\n}());\r\n```\r\n\r\nPossible output:\r\n```\r\nserver log: payload too large\r\nclient got unexpected error: Error: socket hang up\r\n    at connResetException (node:internal/errors:642:14)\r\n    at Socket.socketOnEnd (node:_http_client:486:23)\r\n    at Socket.emit (node:events:381:22)\r\n    at endReadableNT (node:internal/streams/readable:1307:12)\r\n    at processTicksAndRejections (node:internal/process/task_queues:81:21) {\r\n  code: 'ECONNRESET'\r\n}\r\n```\r\n\r\nPossible solution is to not use pipeline but pipe instead. But this requires additional handling like to do `req.unpipe` and `req.resume` to avoid problems for example:\r\n``` js\r\nlet http = require(\"http\");\r\nlet { PassThrough } = require(\"stream\");\r\n\r\nlet server = http.createServer(async function(req, res)\r\n{\r\n\ttry\r\n\t{\r\n\t\tlet len = 0;\r\n\t\tfor await (let chunk of req.pipe(new PassThrough()))\r\n\t\t{\r\n\t\t\tlen += chunk.length;\r\n\t\t\tif(len > 2) throw \"payload too large\";\r\n\t\t}\r\n\t\tres.end(\"ok\");\r\n\t}\r\n\tcatch(err)\r\n\t{\r\n\t\tconsole.log(\"server log:\", err);\r\n\t\treq.unpipe();\r\n\t\treq.resume();\r\n\t\treq.on(\"end\", () => res.end(err));\r\n\t}\r\n});\r\n\r\n(async function()\r\n{\r\n\tawait new Promise(resolve => server.listen(8888, resolve));\r\n\tlet req = http.request({ port: 8888, method: \"post\" });\r\n\treq.end(\"abc\");\r\n\ttry\r\n\t{\r\n\t\tlet res = await new Promise((resolve, reject) => req.on(\"response\", resolve).on(\"error\", reject));\r\n\t\tlet str = \"\";\r\n\t\tfor await (let chunk of res) str += chunk;\r\n\t\tconsole.log(\"client received:\", str);\r\n\t}\r\n\tcatch(err)\r\n\t{\r\n\t\tconsole.log(\"client got unexpected error:\", err);\r\n\t}\r\n\tserver.close();\r\n}());\r\n```\r\n\r\nThis neglected solution with forgotten `unpipe` and `resume` cases EPIPE sometimes or stops after first iteration when using keepalive `Agent`:\r\n\r\n``` js\r\nlet http = require(\"http\");\r\nlet { PassThrough } = require(\"stream\");\r\n\r\nlet server = http.createServer(async function(req, res)\r\n{\r\n\ttry\r\n\t{\r\n\t\tlet len = 0;\r\n\t\tfor await (let chunk of req.pipe(new PassThrough()))\r\n\t\t{\r\n\t\t\tlen += chunk.length;\r\n\t\t\tif(len > 2) throw \"payload too large\";\r\n\t\t}\r\n\t\tres.end(\"ok\");\r\n\t}\r\n\tcatch(err)\r\n\t{\r\n\t\tres.end(err);\r\n\t}\r\n});\r\n\r\n(async function()\r\n{\r\n\tawait new Promise(resolve => server.listen(8888, resolve));\r\n\tlet agent = new http.Agent({ keepAlive: true });\r\n\tlet data = Buffer.alloc(1000000);\r\n\tfor(let i = 0; ; i++)\r\n\t{\r\n\t\tconsole.log(i);\r\n\t\tlet req = http.request({ port: 8888, method: \"post\"/*, agent*/ }).end(data);\r\n\t\tlet res = await new Promise((resolve, reject) => req.on(\"response\", resolve).on(\"error\", reject));\r\n\t\tlet str = \"\";\r\n\t\tfor await (let chunk of res) str += chunk;\r\n\t\tconsole.log(str);\r\n\t}\r\n}());\r\n```\r\n\r\n```\r\n2572\r\npayload too large\r\n2573\r\nnode:internal/process/promises:245\r\n          triggerUncaughtException(err, true /* fromPromise */);\r\n          ^\r\n\r\nError: write EPIPE\r\n    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) {\r\n  errno: -32,\r\n  code: 'EPIPE',\r\n  syscall: 'write'\r\n}\r\n```\r\n\r\n**Describe the solution you'd like**\r\n\r\nPipe is too leak-prone for language which builds on GC. But a fixed and more comfortable pipeline and \"for await\" have unexpected consequences for http streams.\r\n\r\nWould be great to have some options for the `pipeline` function to not destroy streams but instead dump them to the end in case of error (so one can send response when the pipeline ends). Or http request could have the option that `destroy` would just dump stream to end (and only then actually send response) instead of causing EPIPE or similar faults at client side.\r\n","author":{"url":"https://github.com/misos1","@type":"Person","name":"misos1"},"datePublished":"2021-04-16T19:03:20Z","url":"https://github.com/nodejs/node/issues/38262"}