René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Follow curl: Enable setNoDelay() / TCP_NODELAY by default on HTTP(S) 1.x","articleBody":"# Prior art\r\n\r\n## Issues\r\n\r\nThis has been discussed before without any conclusion:\r\n\r\n* 2015-2018 in #906\r\n* 2015 in https://github.com/nodejs/node-v0.x-archive/issues/9235\r\n\r\nRelatedly #906 and the discussion there has been mentioned:\r\n\r\n* Recently, in the beginning of 2020, in #31539 as well as #31541\r\n\r\n## Commits\r\n\r\nThe `setNoDelay()` option was originally introduced in 2009 in https://github.com/nodejs/node/commit/e0ec0036ca2cee7317f63e6462b9631067cf1e38 and was shipped in [v0.1.12](https://github.com/nodejs/node/releases/tag/v0.1.12)\r\n\r\n# Wider context\r\n\r\n## curl\r\n\r\nIn 2016 curl changed it's behavior from being like the one in Node.js to instead be setting `TCP_NODELAY` by default. See https://github.com/curl/curl/commit/4732ca5724072f132876f520c8f02c7c5b654d9 which was released in [v7.60.2](https://curl.haxx.se/mail/archive-2016-09/0013.html).\r\n\r\nMotivation from commit:\r\n\r\n\u003e After a few wasted hours hunting down the reason for slowness during a TLS handshake that turned out to be because of TCP_NODELAY not being set, I think we have enough motivation to toggle the default for this option. We now enable TCP_NODELAY by default and allow applications to switch it off.\r\n\r\n## npm modules\r\n\r\n### agentkeepalive\r\n\r\nThe [agentkeepalive](https://github.com/node-modules/agentkeepalive) module, with ≈3.5 million weekly downloads, pushes its non-configurable default of `socket.setNoDelay(true)` as one of its major benefits.\r\n\r\nIt was added in https://github.com/node-modules/agentkeepalive/commit/c92f5b5bd852c43a8a30700d78c1a02d239d19e4 and references a [Scaling node.js to 100k concurrent connections!](https://blog.caustik.com/2012/04/08/scaling-node-js-to-100k-concurrent-connections/) blog post to explain it.\r\n\r\nUsed by eg. [eggjs/egg](https://github.com/eggjs/egg), [cnpm/cnpmjs.org](https://github.com/cnpm/cnpmjs.org), [pubnub/javascript](https://github.com/pubnub/javascript), [googlemaps/google-maps-services-js](https://github.com/googlemaps/google-maps-services-js), [node-modules/urllib](https://github.com/node-modules/urllib)\r\n\r\n### superagent\r\n\r\nIn 2017 the [superagent](https://github.com/visionmedia/superagent/) module, also with ≈3.5 million weekly downloads, merged https://github.com/visionmedia/superagent/pull/1240 and made `socket.setNoDelay(true)` its non-configurable default, largely arguing that there was no need for any buffering.\r\n\r\n### Other npm modules\r\n\r\nMany modules relies on the default behaviour of Node.js, and thus does not disable any delay by default.\r\n\r\nA quick glance puts all of these in that category, none of them directly calls `setNoDelay` (though they could eg. use something like the agentkeepalive module): `axios`, `bent`, `got`, `node-fetch`, `request`, `unfetch`\r\n\r\n## Documentation\r\n\r\nThe Node.js documentation of `setNoDelay` was recently updated: https://github.com/nodejs/node/pull/31541\r\n\r\nBefore that update, my perception is that many believed that Nagle's algorithm wasn't enabled by defaults, as proved by eg. the conversation in superagent: https://github.com/visionmedia/superagent/pull/1240\r\n\r\n# Proposed change\r\n\r\nTo make `TCP_NODELAY` / `setNoDelay` the default for HTTP(S) 1.x + maybe introduce a `setDelay()` method to disable it.\r\n\r\nProbably in a new major version, as it can be a breaking change in some contexts.\r\n\r\n## Why \r\n\r\nI believe that the expectations has changed since `setNoDelay` was introduced in 2009.\r\n\r\nWith eg. curl now setting it by default I believe most expects `TCP_NODELAY` / `setNoDelay` to be set by default and thus Nagle's algorithm to be disabled by default.\r\n\r\nThe confusion the current default seems to make in issues like https://github.com/visionmedia/superagent/pull/1240, especially before #31541, supports that.\r\n\r\nAlso: As `TCP_NODELAY` / Nagle's algorithm is implemented at an OS level this can also have a minor added benefit of improved cross-platform behavior, as no platform will add a delay.\r\n\r\n## Benefits\r\n\r\n* Better comply with general expectations among developers\r\n* Move with the times and in sync with other major players like `curl`\r\n* In some cases better performance due to no delay. Especially so in real time scenarios.\r\n\r\n## Downsides\r\n\r\n* Possibly increased bandwidth use\r\n* Possibly decreased performance due to bandwidth saturation\r\n* As any change of a default on this level, it breaks the expectations of those who knowingly wants the current default behaviour.\r\n\r\n# My background\r\n\r\nI have no expertise in the implementation of Nagle's algorithm or the TCP-stack in general, though I became intrigued by this and did my research and through that research I found that most were either confused by this or had already changed the default for this.\r\n\r\nComments, suggestions, feedback and link to other discussions would be most helpful.","author":{"url":"https://github.com/voxpelli","@type":"Person","name":"voxpelli"},"datePublished":"2020-07-03T16:51:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":12},"url":"https://github.com/34185/node/issues/34185"}