René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"smart-search returns \"Invocation stopped\" on a large store while the worker keeps re-registering","articleBody":"\nMeasurements taken 2026-10-19 to 2026-10-20 on one macOS machine. No secrets, hostnames, user data or memory content are included here; redacted `daemon.log` excerpts and the `/health` `functionMetrics` block are available on request.\n\n## Summary\n\nOn a large local store, `POST /agentmemory/smart-search` fails with HTTP 500 `Invocation stopped` after 6 to 12 seconds, even though the daemon log shows the same search finishing with results. The worker connection to the iii engine drops and re-registers every few minutes, which stops in-flight invocations. The same behaviour appeared on 0.9.28 (engine 0.11.2) and on 0.9.30 (engine 0.22.1).\n\n## Environment\n\n- macOS (Darwin 27), Node v24.18.0, run as a launchd service with KeepAlive.\n- agentmemory 0.9.28 with iii 0.11.2; tested again on 0.9.30 with iii 0.22.1, then rolled back.\n- Store: about 2.0 GB, about 9,000 files under the data directory. Roughly 900 sessions and 50 concurrent MCP proxy processes from separate editor sessions.\n- LLM calls go through a local OpenAI-compatible bridge to a Claude model. Consolidation, graph extraction and reflect were enabled, then turned off to cut load.\n\n## What we saw\n\n1. **`smart-search` 500.** `{\"error\":\"Invocation stopped\"}` (0.9.30: `{\"error\":{\"code\":\"invocation_stopped\",...}}`) after 6 to 12 s. The daemon log for the same query shows `Smart search compact {\"results\":5,\"lessons\":5}`. At other moments the same route returns `Function api::smart-search not found` or 404, which matches the worker being between registrations.\n2. **Worker reconnect churn.** `[iii] Reconnecting in ~1000ms (attempt 1)` followed by `Worker registered with ID: \u003cnew id>` repeated; roughly 6 re-registrations in 15 to 80 minutes on a quiet daemon, dozens per hour under load.\n3. **Invocation timeouts.** `mem::lesson-recall` and `mem::reflect` hit the 180 s bound; `state::set` and `state::list` also timed out during graph extraction. `mem::summarize` shows 10,847 failures of 31,854 calls in `/health` functionMetrics.\n4. **`/health` returns 503 when CPU is high** (alert `cpu_critical`), although the daemon is working. A supervisor that polls `/health` will kill a busy but healthy daemon. `/livez` is fine.\n5. **Index persistence warnings after hard kills.** `index persistence: BM25 shard missing` and `vector shard missing` repeat. After the supervisor force-killed the process about every 11 minutes, newly saved memories were stored (`GET /memories/:id` returns 200) but did not appear in search.\n6. **0.9.30 specifics.**\n   - The CLI waits a fixed 15 s for the worker (`waitForAgentmemoryReady(15e3)`), then exits 1. On a store this size the worker needs longer, so the supervisor restarts it in a loop. Raising the constant to 240 s stopped the loop but not item 1.\n   - REST auth is on by default. Older MCP proxy processes that were already running do not send the secret, and the memory tools then return empty results instead of an error, which looks like \"no memories\" rather than \"unauthorized\".\n   - `MAX_OBS_PER_SESSION=200` moves captures from long sessions to dead letters (`Session observation limit reached (200)`).\n\n## What we tried\n\n- Replaced `/health` with `/livez` in the supervisor: restarts stopped, search still fails.\n- Disabled consolidation, graph extraction and reflect: load dropped, search still fails.\n- Upgraded 0.9.28 to 0.9.30 (engine 0.22.1): search still fails; worker restarted 8 times in 40 minutes; rolled back.\n\n## Questions\n\n1. Is `invocation_stopped` expected when the worker re-registers mid-call, and what makes the worker connection drop on a busy event loop (ping timeout, message size, something else)?\n2. Is there a setting to raise the worker-ready wait, the invocation timeout, or the engine ping interval?\n3. Can the MCP proxy surface a 401 or 500 to the caller instead of returning an empty result list?\n4. Can `/health` report 200 with a degraded flag instead of 503 under load, so supervisors do not kill a working daemon?\n\n## Possibly related\n\n#1378 (worker heartbeat death loop on a large store), #866 (15 s MCP proxy call timeout on large stores), #890 (`Invocation stopped` on large payloads). The behaviour here may share a cause with #1378; happy to test a fix.\n","author":{"url":"https://github.com/chanfox7","@type":"Person","name":"chanfox7"},"datePublished":"2026-10-20T06:23:09Z","url":"https://github.com/rohitg00/agentmemory/issues/1593"}