René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"quic: Current implementation results in tiny Uint8Arrays in streaming mode","articleBody":"@jasnell \nHere is a test (I just wanted to trigger some problem I have seen when I create my PR (I could not provoke any problem, well I have seen some stalls, but they are gone, maybe I did something wrong or net load changed on my machine)):\n```\n// Flags: --experimental-quic --experimental-stream-iter --no-warnings\n\n// Test: bidirectional stream echo with binary.\n// The client sends a message, the server reads it and echoes it back.\n// Both directions of the bidi stream carry data and are properly FIN'd.\n// Verifies that both client and server can read and write on the same stream.\n\nimport { hasQuic, skip, mustCall } from '../common/index.mjs';\nimport assert from 'node:assert';\n\nconst { strictEqual } = assert;\n\nif (!hasQuic) {\n  skip('QUIC is not enabled');\n}\n\nconst { listen, connect } = await import('../common/quic.mjs');\nconst { bytes, drainableProtocol: dp } = await import('stream/iter');\n\nconst chunkSizes = [60000, 12, 1000000, 50000, 1600, 20000, 1000000, 30000, 0, 100]\nconst numChunks = chunkSizes.length\nconst byteLength = chunkSizes.reduce((accumulator, currentValue) =\u003e accumulator + currentValue, 0);\n\n\nconst sleep = ms =\u003e new Promise(res =\u003e setTimeout(res, ms));\n\n// Build a deterministic payload so we can verify integrity.\nfunction buildChunk(index) {\n  const chunk = new Uint8Array(chunkSizes[index]);\n  // Fill with a pattern derived from the chunk index.\n  const val = index \u0026 0xff;\n  for (let i = 0; i \u003c chunkSizes[index]; i++) {\n    chunk[i] = (val + i) \u0026 0xff;\n  }\n  return chunk;\n}\n\nfunction checksum(data) {\n  let sum = 0;\n  for (let i = 0; i \u003c data.byteLength; i++) {\n    sum = (sum + data[i]) | 0;\n  }\n  return sum;\n}\n\n// Compute expected checksum.\nlet expectedChecksum = 0;\nfor (let i = 0; i \u003c numChunks; i++) {\n  const chunk = buildChunk(i);\n  expectedChecksum = (expectedChecksum + checksum(chunk)) | 0;\n}\n\nconst done = Promise.withResolvers();\n\nconst serverEndpoint = await listen(mustCall((serverSession) =\u003e {\n  serverSession.onstream = mustCall(async (stream) =\u003e {\n    const writer = stream.writer\n    let written = 0;\n    for await (const chunks of stream) {\n      for (const chunk of chunks) {\n        written += chunk.byteLength;\n        console.log('chunk', chunk.byteLength)\n\n        while (!writer.writeSync(chunk)) {\n          // Flow controlled — wait for drain before retrying.\n          const drainable = writer[dp]();\n          if (drainable) await drainable;\n        }\n      }\n    }\n    writer.end();\n\n    // await stream.closed; // may be we are not closing\n    serverSession.close();\n    done.resolve();\n  });\n}));\n\nconst clientSession = await connect(serverEndpoint.address);\nawait clientSession.opened;\n\n\nconst stream = await clientSession.createBidirectionalStream();\n\nconst readChunks = [];\nconst readFromStream = async (stream) =\u003e {\n  for await (const chunks of stream) {\n    readChunks.push(...chunks);\n  }\n  const receivedBytes = readChunks.reduce((accu, curVal) =\u003e accu + curVal.byteLength, 0);\n\n  strictEqual(receivedBytes, byteLength);\n\n  let receivedChecksum = 0;\n  for (const chunk of readChunks) {\n    receivedChecksum = (receivedChecksum + checksum(chunk)) | 0;\n  }\n\n  strictEqual(receivedChecksum, expectedChecksum);\n}\n\n\nconst writeToStream = async (stream) =\u003e {\n  const w = stream.writer\n  for (let i = 0; i \u003c numChunks; i++) {\n    const chunk = buildChunk(i);\n    while (!w.writeSync(chunk)) {\n      // Flow controlled — wait for drain before retrying.\n      const drainable = w[dp]();\n      if (drainable) await drainable;\n    }\n    await sleep(1)\n  }\n  w.endSync();\n}\n\nawait Promise.all([readFromStream(stream), writeToStream(stream)])\n\n\n\n\nawait Promise.all([stream.closed, done.promise]);\nawait clientSession.close();\nawait serverEndpoint.close();\n```\nWhat you see here is that the chunks that arrive are limited by UDP packet size:\n```\nchunk 1000\nchunk 1156\nchunk 1156\nchunk 1156\nchunk 1156\nchunk 1156\nchunk 1156\nchunk 1156\nchunk 1156\n.......\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 6\nchunk 5664\nchunk 8192\nchunk 3439\nchunk 1141\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1154\nchunk 1154\n.......\nchunk 1154\nchunk 1144\nchunk 506\nchunk 8192\nchunk 8192\nchunk 8192\nchunk 3705\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1147\nchunk 4608\nchunk 8192\nchunk 2167\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1146\nchunk 4993\nchunk 8192\nchunk 1804\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\n.....\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 3496\nchunk 8192\nchunk 8192\nchunk 8192\nchunk 8192\nchunk 652\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1145\nchunk 5355\nchunk 8192\nchunk 3742\nchunk 1143\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 4\nchunk 4571\nchunk 8192\nchunk 8192\nchunk 7877\nchunk 1143\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\n......\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1148\nchunk 1154\nchunk 1154\nchunk 1154\n.......\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1147\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1153\nchunk 1153\nchunk 1153\nchunk 1148\nchunk 1154\nchunk 1154\nchunk 1154\n.......\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1145\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\n.......\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 1154\nchunk 3\nchunk 100\n```\n\nThis is not astonishing by the current design of the underlying node_blob. However, I believe that this can result in poor performance, as the overhead per packet is really high and it may lead to memory fragmentation.\n(see https://github.com/nodejs/node/pull/60237, for example:\nhttps://github.com/nodejs/node/pull/60237/changes/78b724d9e2d9ab6f6fa425d23087966d869bd0fd\nwhich does this early on in the code.\n(I can rebase and rework this part if desired).\n\nSideload: I will try to create some other tests to trigger some cases I know from other implementations. \nBut so far it is really working great. So I may soon move to implement Webtransport on the js side on top, and see what is missing, and run tests against browsers. (Though on the long run a complete native solution is desired.).\n\n\n\n","author":{"url":"https://github.com/martenrichter","@type":"Person","name":"martenrichter"},"datePublished":"2026-05-14T09:41:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/63303/node/issues/63303"}