René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Worker drops its iii connection when an endpoint response exceeds 16 MiB (worker→engine frame cap); semantic/insights provenance lists are unbounded","articleBody":"## Summary\n\nOnce a state scope grows past **16 MiB**, any HTTP endpoint that returns the whole scope (e.g. `GET /agentmemory/semantic`) makes the **worker drop its connection to iii-engine**. The worker then reconnects, and every endpoint returns 404/500 until it is back. With a dashboard polling, this becomes a reconnect loop: we measured **280 reconnects in 10 minutes**.\n\nThe usual reading is \"the worker↔engine frame is capped at 16 MiB and not configurable\", which leads to purging big scopes. We measured both directions, and the cap only applies to **messages the worker sends to the engine**. Messages the engine sends to the worker go through up to ~100 MiB. Internal `kv.list()` calls are therefore fine. What breaks is an **endpoint response** that embeds a whole scope.\n\nTwo unbounded provenance lists make scopes reach that size quickly. Details and proposed fixes below.\n\n## Environment\n\n- `@agentmemory/agentmemory` **0.9.29** (global install, server = `dist/cli.mjs` → `dist/src-\u003chash>.mjs`)\n- `iii` engine **0.11.2**, `iii-sdk` 0.11.2 (bundled), state adapter `kv` / `file_based`\n- Node 22, Linux x86_64\n\nStore sizes at the time: `mem:insights` 159 MB, `mem:semantic` 34.8 MB, `mem:audit` 29.3 MB.\n\n## Reproduction (both directions, against a live engine, using a throwaway scope)\n\n```js\n// repro.mjs — run with: node repro.mjs \u003cmode>\nimport { registerWorker } from \"\u003cagentmemory>/node_modules/iii-sdk/dist/index.mjs\";\nconst mode = process.argv[2];\nconst sdk = registerWorker(\"ws://127.0.0.1:49134\", { workerName: \"repro-\" + mode, invocationTimeoutMs: 60000 });\nconst t0 = Date.now();\nconst done = (m) => { console.log(mode, m, Date.now() - t0 + \"ms\"); process.exit(0); };\ntry {\n  if (mode === \"list\") {                       // engine -> worker\n    const r = await sdk.trigger({ function_id: \"state::list\", payload: { scope: \"mem:audit\" } });\n    done(\"OK bytes=\" + JSON.stringify(r).length);\n  } else {                                     // worker -> engine, e.g. mode \"set-17\"\n    const n = Number(mode.split(\"-\")[1]) * 1024 * 1024;\n    await sdk.trigger({ function_id: \"state::set\", payload: { scope: \"repro:frame\", key: \"k\", value: { x: \"a\".repeat(n) } } });\n    await sdk.trigger({ function_id: \"state::delete\", payload: { scope: \"repro:frame\", key: \"k\" } });\n    done(\"OK\");\n  }\n} catch (e) { done(\"ERROR \" + (e?.message ?? e)); }\n```\n\nResults:\n\n| Direction | Payload | Result |\n|---|---|---|\n| engine → worker, `state::list` of `mem:audit` | 27.7 MB (66,381 items) | **OK**, 0.9 s |\n| engine → worker, `state::list` of a test scope | 84 MiB | **OK**, 0.4 s |\n| engine → worker, `state::list` of a test scope | 112 MiB | **fails**: `Invocation timeout after 90000ms` |\n| worker → engine, `state::set` | 15 MiB | **OK** |\n| worker → engine, `state::set` | 17 MiB | **fails**: connection dropped silently, `Invocation timeout after 60000ms` |\n\nThe running agentmemory worker was not affected by these tests. Only the throwaway worker's own connection dropped.\n\n## Root cause\n\n- **worker → engine:** the engine serves the worker WebSocket through axum (`engine/src/workers/worker/ws_handler.rs`). This looks like tungstenite's default `max_frame_size` of 16 MiB. The Node `ws` client sends each message as a single frame, so any message over 16 MiB is rejected, the socket closes, and the pending invocation times out. Nothing in `iii-config.yaml` exposes this setting.\n- **engine → worker:** the Node `ws` client uses its default `maxPayload` of 100 MiB (`iii-sdk` does not override it). Reads up to that size succeed.\n- **What actually sends > 16 MiB:** an HTTP handler such as `api::semantic-list` returns `{ semantic: await kv.list(KV.semantic) }`. That return value travels **worker → engine** (to iii-http), so it hits the cap. The same applies to the export endpoint, which returns memories, actions, semantic, procedural, relations and graph together. 130 `api::*` handlers are registered, and several return whole scopes.\n\n## Why scopes get that big: two unbounded provenance lists\n\n1. **`semantic.sourceSessionIds`**, in `mem::consolidate-pipeline`:\n   ```js\n   sourceSessionIds: recentSummaries.map((s) => s.sessionId),\n   ```\n   `recentSummaries` is the latest batch of up to 20 session summaries. **The same 20 ids are copied into every new fact** of that pass, so this is not real per-fact provenance. In our store this field was **69 % of `mem:semantic`'s bytes**.\n2. **`insights.sourceMemoryIds`**, in `mem::reflect`:\n   ```js\n   sourceMemoryIds: cluster.factIds,\n   ```\n   This holds every semantic fact sharing a term with the cluster and is never capped (same for `sourceLessonIds` / `sourceCrystalIds`). `mem:insights` reached **159 MB**.\n   Also, `insight-decay-sweep` only rewrites insights whose confidence changed. Insights at the floor, or soft-deleted, are never rewritten, so bounding at write time alone would not shrink the existing scope.\n\nAs far as we can tell, nothing in the 0.9.29 bundle reads these fields back, except export/import and mesh sync, which copy them as-is. When we checked the 0.9.30 tarball, both assignments were unchanged. `boundSources` (`slice(-32)`) in 0.9.30 only applies to graph `sourceObservationIds`.\n\n## What we deployed locally (works, measured)\n\n**1. Paginate `api::*` responses above a byte budget.** We wrap `sdk.registerFunction` once, right after `registerWorker`:\n\n```js\n{\n  const budget = Number(process.env.AGENTMEMORY_REPONSE_MAX_OCTETS) || 12 * 1024 * 1024;\n  const size = (v) => Buffer.byteLength(JSON.stringify(v) ?? \"\");\n  const int = (v) => { const n = Number.parseInt(v, 10); return Number.isFinite(n) && n >= 0 ? n : null; };\n  const paginate = (req, res) => {\n    if (!res?.body || typeof res.body !== \"object\" || Array.isArray(res.body)) return res;\n    const q = req?.query_params || {};\n    const asked = int(q.offset) !== null || int(q.limit) !== null;\n    const total = size(res.body);\n    if (!asked && total \u003c= budget) return res;\n    const arrays = Object.keys(res.body).filter((k) => Array.isArray(res.body[k]))\n      .map((k) => [k, size(res.body[k])]).sort((a, b) => b[1] - a[1]);\n    if (!arrays.length) return res;\n    const offset = int(q.offset) ?? 0;\n    const [bigName, bigBytes] = arrays[0];\n    let limit = int(q.limit) || Math.max(1, Math.floor((budget * 0.9 - (total - bigBytes)) / (bigBytes / (res.body[bigName].length || 1))));\n    const orig = Object.fromEntries(arrays.map(([k]) => [k, res.body[k]]));\n    let fields;\n    for (;;) {\n      fields = [];\n      for (const [k] of arrays) {\n        res.body[k] = orig[k];\n        if (!asked && size(res.body) \u003c= budget) break;\n        res.body[k] = orig[k].slice(offset, offset + limit);\n        fields.push(k);\n      }\n      if (size(res.body) \u003c= budget || limit === 1) break;\n      limit = Math.max(1, Math.floor(limit / 2));\n    }\n    const totals = Object.fromEntries(fields.map((k) => [k, orig[k].length]));\n    const more = fields.some((k) => offset + limit \u003c orig[k].length);\n    res.body.pagination = { fields, offset, limit, totals, next_offset: more ? offset + limit : null };\n    return res;\n  };\n  const register = sdk.registerFunction.bind(sdk);\n  sdk.registerFunction = (id, handler, ...rest) => {\n    if (typeof id !== \"string\" || !id.startsWith(\"api::\") || typeof handler !== \"function\") return register(id, handler, ...rest);\n    return register(id, async (req, ...a) => {\n      const res = await handler(req, ...a);\n      try { return paginate(req, res); } catch { return res; }\n    }, ...rest);\n  };\n}\n```\n\n- The response shape is unchanged below the budget.\n- Above it, the largest arrays are paged with `?offset=&limit=` and the body carries `pagination.next_offset`.\n- A client that ignores pagination gets the first page instead of taking the worker down.\n\nMeasured in production: `GET /agentmemory/semantic` (15.7 MB) is served as 2 pages (20,861 + 8,087 facts). All 28,948 facts were rebuilt with no duplicates, in 1.8 s, with 0 worker reconnects. The effective per-scope ceiling moves from 16 MiB to ~100 MiB, the engine → worker limit.\n\n**2. Bound the provenance lists at the single `state::set` choke point** (`StateKV.set`):\n\n- `mem:semantic`: keep the 3 most recent `sourceSessionIds` and add `sourceSessionCount` with the original length.\n- `mem:insights`: keep the last 32 of `sourceMemoryIds` / `sourceLessonIds` / `sourceCrystalIds`.\n\nThe decay tier of `consolidate-pipeline` rewrites every semantic fact on each pass. The existing scope therefore shrank by itself within one pass: **34.8 MB → 16.5 MB**, 28,956 facts rewritten, 0 errors. `mem:insights` had to be purged once (service stopped, file backed up). It regenerated to 100 KB.\n\n## Suggested upstream fixes\n\n1. **Paginate list endpoints** (`/agentmemory/semantic`, `/memories`, export, …): add `offset`/`limit` (or a cursor) and return `next_offset`, at least whenever the serialized body would exceed a budget below the frame cap. The viewer would need to follow `next_offset`.\n2. **Do not copy the batch's session ids into every semantic fact.** Either record the sessions a fact actually came from, or store a count. If the list stays, cap it the way `boundSources` caps graph sources.\n3. **Cap `insights.source*Ids`** (`slice(-32)`, like `boundSources`). Also give soft-deleted and floor-confidence insights a hard-delete or compaction path; today nothing ever rewrites or removes them.\n4. **Engine/SDK side, optional:** expose the WebSocket `max_frame_size` / `max_message_size` in the iii config, or have the SDK fragment large messages. Tungstenite reassembles fragmented messages up to `max_message_size`, which defaults to 64 MiB. Either way, failing loudly instead of a silent drop and a 60 s timeout would have saved us a lot of diagnosis.\n\nHappy to open a PR for (1)–(3) against the TypeScript sources if that direction suits you.\n","author":{"url":"https://github.com/Fguarien","@type":"Person","name":"Fguarien"},"datePublished":"2026-10-17T19:46:48Z","url":"https://github.com/rohitg00/agentmemory/issues/1579"}