René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"async iterators/generators in stream.pipeline()","articleBody":"Apologies for all the code here.\r\n\r\nWe can consume a `Readable` stream using an async iterator:\r\n\r\n```js\r\n// source: http://2ality.com/2018/04/async-iter-nodejs.html\r\nasync function printAsyncIterable(iterable) {\r\n  for await (const chunk of iterable) {\r\n    console.log('\u003e\u003e\u003e '+chunk);\r\n  }\r\n}\r\n\r\nprintAsyncIterable(fs.createReadStream('my-file.txt', 'utf8'));\r\n```\r\n\r\nAnd we can use async generators similarly to how one would use a `Transform` stream: \r\n\r\n```js\r\n/**\r\n * Parameter: async iterable of chunks (strings)\r\n * Result: async iterable of lines (incl. newlines)\r\n */\r\nasync function* chunksToLines(chunksAsync) {\r\n  let previous = '';\r\n  for await (const chunk of chunksAsync) {\r\n    previous += chunk;\r\n    let eolIndex;\r\n    while ((eolIndex = previous.indexOf('\\n')) \u003e= 0) {\r\n      // line includes the EOL\r\n      const line = previous.slice(0, eolIndex+1);\r\n      yield line;\r\n      previous = previous.slice(eolIndex+1);\r\n    }\r\n  }\r\n  if (previous.length \u003e 0) {\r\n    yield previous;\r\n  }\r\n}\r\n\r\n/**\r\n * Parameter: async iterable of lines\r\n * Result: async iterable of numbered lines\r\n */\r\nasync function* numberLines(linesAsync) {\r\n  let counter = 1;\r\n  for await (const line of linesAsync) {\r\n    yield counter + ': ' + line;\r\n    counter++;\r\n  }\r\n}\r\n```\r\n\r\nThen, we can \"pipe\" these together like so:\r\n\r\n```js\r\nasync function main() {\r\n  printAsyncIterable(\r\n    numberLines(\r\n      chunksToLines(\r\n        fs.createReadStream('my-file.txt', 'utf8')\r\n      )\r\n    )\r\n  );\r\n}\r\nmain();\r\n```\r\n\r\nThat's neat, but also kind of hideous.  What if we could leverage `stream.pipeline()` to do something like this?\r\n\r\n```js\r\nasync function main() {\r\n  stream.pipeline(\r\n    fs.createReadStream('my-file.txt', 'utf8'),\r\n    chunksToLines,\r\n    numberLines,\r\n    printAsyncIterable\r\n  );\r\n}\r\nmain();\r\n```\r\n\r\nI'm unfamiliar with the guts of `stream.pipeline()`--and completely new to async iterators and generators--so don't know how feasible something like this is.\r\n\r\nFWIW, the \"hideous nested function calls\" can be naively replaced by use of the godlike `Array.prototype.reduce()`:\r\n\r\n```js\r\nconst pipeline = async (...args) =\u003e args.reduce((acc, arg) =\u003e arg(acc));\r\n\r\nasync function main() {\r\n  pipeline(\r\n    fs.createReadStream('my-file.txt', 'utf8'),\r\n    chunksToLines,\r\n    numberLines,\r\n    printAsyncIterable\r\n  );\r\n}\r\nmain();\r\n```\r\n\r\nReference: https://twitter.com/b0neskull/status/1115325542566227968","author":{"url":"https://github.com/boneskull","@type":"Person","name":"boneskull"},"datePublished":"2019-04-08T20:23:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":10},"url":"https://github.com/27140/node/issues/27140"}