René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Expose and write headers on 1xx intermediate status codes","articleBody":"Background: I'm currently working on a document for standardizing how servers may communicate intermediate status of a long-running operation over HTTP (operations running minutes to days). Part of this involves use of 1xx status codes, available in HTTP/1.1 and HTTP/2.\r\n\r\nHTTP specifies that a request may have multiple 1xx responses before a final 2xx-5xx response. These responses, like any other, may have headers:`101 Switching Protocols`, `102 Processing`, and `103 Early Hints` are all known to use headers to convey additional information about the intermediate status. (For example, 101 uses [Upgrade](https://httpwg.org/specs/rfc7230.html#header.upgrade), 102 uses [Status-URI](https://tools.ietf.org/html/rfc2518#section-10.1), and 103 uses [Link](https://httpwg.org/specs/rfc8297.html)).\r\n\r\nThere is currently seemingly no way to write or read these headers. Given the definition of 1xx (which is mandatory in HTTP/1.1 and HTTP/2), I would expect to be able to call `ServerResponse#writeHead` multiple times with a 1xx status code, however, this does not flush the headers to the response, and appears to cause an error to be thrown once the final headers are written! Node.js added `ServerResponse#writeProcessing` in v10.0.0, however, this is specific to a single status code, does not support headers, and is not forward compatible with future status codes.\r\n\r\nI would also expect the `ClientRequest#on(\"information\")` event to include a headers object, so I can read this data.\r\n\r\nThe only known workarounds are to use `ServerResponse#_writeRaw` which is not a public API; and the only available option for clients is to parse responses manually.\r\n\r\n---\r\n\r\nHere is a script demonstrating the expected behavior:\r\n\r\n```javascript\r\n\r\nconst http = require('http');\r\nconst server = http.createServer(handleRequest);\r\nserver.listen(0);\r\nconsole.error('Listening on port '+server.address().port);\r\n\r\nfunction handleRequest(req, res){\r\n    var tasks = [\r\n        'Herding cats',\r\n        'Digging holes',\r\n        'Filling in holes',\r\n        'Making tea',\r\n    ];\r\n    function writeProgress(i){\r\n        console.error('writeProgress('+i+')');\r\n        if(false){\r\n            res.writeHead(102, 'Processing', {\r\n                'Progress': `${i}/${tasks.length} (${tasks[i]})`,\r\n            });\r\n        }else{\r\n            res._writeRaw('HTTP/1.1 102 Processing\\r\\n');\r\n            res._writeRaw('Progress: '+i+'/'+tasks.length+' ('+tasks[i]+')\\r\\n');\r\n            res._writeRaw('\\r\\n');\r\n        }\r\n    }\r\n    function next(){\r\n        if(++i===tasks.length){\r\n            res.setHeader('Content-Type', 'text/plain');\r\n            res.end('All done!\\r\\n');\r\n        }else{\r\n            writeProgress(i);\r\n            setTimeout(next, 100);\r\n        }\r\n    }\r\n    var i = 0;\r\n    writeProgress(i);\r\n    setTimeout(next, 100);\r\n}\r\n\r\nvar req = http.request({\r\n    host: server.address().address,\r\n    port: server.address().port,\r\n    path: '/',\r\n});\r\nreq.end();\r\nreq.on('information', function(res){\r\n    console.log(res);\r\n});\r\nreq.on('response', function(res){\r\n    res.pipe(process.stdout);\r\n    res.on('end', console.error);\r\n});\r\n\r\n```\r\n\r\nWhen `_writeRaw` is used, the Node.js client sees only the `statusCode`:\r\n\r\n```\r\n$ node -v\r\nv12.1.0\r\n$ node demo.js \r\nListening on port 49213\r\nwriteProgress(0)\r\n{ statusCode: 102 }\r\nwriteProgress(1)\r\n{ statusCode: 102 }\r\nwriteProgress(2)\r\n{ statusCode: 102 }\r\nwriteProgress(3)\r\n{ statusCode: 102 }\r\nAll done!\r\n```\r\n\r\nWhen `_writeRaw` is used, `curl` is able to parse the response as expected:\r\n\r\n```\r\n$ curl -v http://localhost:49213/\r\n*   Trying ::1...\r\n* TCP_NODELAY set\r\n* Connected to localhost (::1) port 49213 (#0)\r\n\u003e GET / HTTP/1.1\r\n\u003e Host: localhost:49213\r\n\u003e User-Agent: curl/7.54.0\r\n\u003e Accept: */*\r\n\u003e \r\n\u003c HTTP/1.1 102 Processing\r\n\u003c Progress: 0/4 (Herding cats)\r\n\u003c HTTP/1.1 102 Processing\r\n\u003c Progress: 1/4 (Digging holes)\r\n\u003c HTTP/1.1 102 Processing\r\n\u003c Progress: 2/4 (Filling in holes)\r\n\u003c HTTP/1.1 102 Processing\r\n\u003c Progress: 3/4 (Making tea)\r\n\u003c HTTP/1.1 200 OK\r\n\u003c Content-Type: text/plain\r\n\u003c Date: Sun, 26 May 2019 21:02:58 GMT\r\n\u003c Connection: keep-alive\r\n\u003c Content-Length: 11\r\n\u003c \r\nAll done!\r\n* Connection #0 to host localhost left intact\r\n```\r\n\r\nHowever, when `writeHead` is used, the data never makes it out of the socket, and an error is emitted towards the end:\r\n\r\n```\r\n$ node -v\r\nv12.1.0\r\n$ node demo.js \r\nListening on port 49224\r\nwriteProgress(0)\r\nwriteProgress(1)\r\nwriteProgress(2)\r\nwriteProgress(3)\r\n_http_outgoing.js:467\r\n    throw new ERR_HTTP_HEADERS_SENT('set');\r\n    ^\r\n\r\nError [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\r\n    at ServerResponse.setHeader (_http_outgoing.js:467:11)\r\n    at Timeout.next [as _onTimeout] (.../demo.js:28:17)\r\n    at listOnTimeout (internal/timers.js:531:17)\r\n    at processTimers (internal/timers.js:475:7)\r\n\r\n$ curl -v http://localhost:49226/\r\n*   Trying ::1...\r\n* TCP_NODELAY set\r\n* Connected to localhost (::1) port 49226 (#0)\r\n\u003e GET / HTTP/1.1\r\n\u003e Host: localhost:49226\r\n\u003e User-Agent: curl/7.54.0\r\n\u003e Accept: */*\r\n\u003e \r\n* Empty reply from server\r\n* Connection #0 to host localhost left intact\r\ncurl: (52) Empty reply from server\r\n```\r\n\r\n---\r\n\r\nIn summary:\r\n\r\n* `ClientRequest#on(\"information\")` should expose a res-like object with headers, and\r\n* `ServerResponse#writeHead` should allow multiple calls with a 1xx status code that is immediately flushed to the socket.","author":{"url":"https://github.com/awwright","@type":"Person","name":"awwright"},"datePublished":"2019-05-26T21:08:09.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":12},"url":"https://github.com/27921/node/issues/27921"}