René's URL Explorer Experiment

Title: feat(runtime-core,mcp): keep deepnote-toolkit servers warm across MCP runs by jamesbhobbs · Pull Request #520 · deepnote/deepnote · GitHub

Open Graph Title: feat(runtime-core,mcp): keep deepnote-toolkit servers warm across MCP runs by jamesbhobbs · Pull Request #520 · deepnote/deepnote

X Title: feat(runtime-core,mcp): keep deepnote-toolkit servers warm across MCP runs by jamesbhobbs · Pull Request #520 · deepnote/deepnote

Description: Third in the chain: stacked on #519 (runtime hardening), which is stacked on #518 (interpreter resolution). Merge those first. Part of #162. Problem Every deepnote_run tool call in the MCP spawns a deepnote-toolkit server (a Python process that imports Jupyter and the toolkit), waits for it, starts a kernel, runs, and kills everything. That is several seconds of overhead on every agent step, and none of it is the kernel work the agent asked for. Starting a kernel session on a running server is cheap; starting the server is what costs. Change ServerPool (@deepnote/runtime-core). Keeps one toolkit server warm per interpreter, working directory, port and environment. Callers acquire() a lease and release() it; concurrent acquires for the same key share one startup; a server with no lease is stopped after idleTimeoutMs (default 5 min, 0 stops immediately). shutdown() stops everything and rejects later acquires; killAll() is the synchronous last resort for process.on('exit') and signals the recorded Jupyter and language-server children before the supervisor. ExecutionEngine attaches to a provided server. new ExecutionEngine(config, { server }) starts its own kernel on an existing server and, on stop(), disconnects only the kernel. The engine watches the server with a removable process listener that it detaches on stop, so engines attached to a long-lived server do not accumulate. MCP. deepnote_run leases a warm server for each call and attaches a fresh engine, so every call still gets a fresh kernel and no variables leak between calls. DEEPNOTE_MCP_SERVER_IDLE_SECONDS tunes the idle timeout. Pooled servers are stopped on SIGINT/SIGTERM and when the MCP transport closes, and killed on exit, so no toolkit server outlives its host. Review findings folded in Three issues raised against the earlier combined PR are fixed here with tests against real Jupyter: Concurrent runs could share a kernel. KernelClient always requested the session path deepnote-cli, and Jupyter returns the existing session for a known path, so two engines on one server got the same kernel and either cleanup shut it down. Each connection now uses a unique session path. An integration test runs two engines concurrently on one server and asserts neither sees the other's variables. A server whose Jupyter had crashed stayed reusable. The pool only evicted on supervisor exit, and the supervisor survives a dead Jupyter child. acquire() now health-checks a warm server before handing it out and replaces one that no longer answers. An integration test kills Jupyter under a pooled server and asserts the next acquire gets a fresh server and the old supervisor is stopped. An unexpectedly exited supervisor left its children behind. Eviction on exit now runs the same child cleanup as stopServer, so a Jupyter or language server that outlived its supervisor is terminated. Further findings on the pool's edge cases are fixed here as well: in-flight stops are tracked, so shutdown() waits for a server that is already being stopped (and tolerates one that fails to stop) and killAll() can still signal it; acquire() rechecks that the pool is open after its health probe and replacement stop, so a shutdown during the probe cannot start and abandon a replacement; only idle servers are probed, so a false alarm can never stop a server from under a run that holds it; the MCP releases its lease even when engine.stop() rejects; and DEEPNOTE_MCP_SERVER_IDLE_SECONDS is capped at the Node timer limit so an oversized value cannot make the idle timer fire at once. Tests Unit: pool (reuse, shared startup, idle stop, immediate stop, eviction on exit with cleanup, health-check replacement, startup failure, shutdown semantics, killAll child signalling), engine (attach to a provided server, no listener accumulation), kernel client (unique session paths), MCP (pooled runs, lease release on failure, idle env var parsing). Integration (pnpm test:integration): concurrent engines on one server; pool recovery after a Jupyter crash, with the leak guard confirming nothing survives. Docs packages/mcp/README.md: warm servers and DEEPNOTE_MCP_SERVER_IDLE_SECONDS. 🤖 Generated with Claude Code Summary by CodeRabbit New Features MCP executions reuse warm servers for faster repeated runs while creating a fresh kernel for each call. Healthy servers can be shared by concurrent executions, while unresponsive servers are automatically replaced. Configure server idle duration with DEEPNOTE_MCP_SERVER_IDLE_SECONDS; the default is five minutes. Separate connections now receive unique sessions to prevent interference. Execution timeouts now apply to agent activity and dynamically run code. Bug Fixes Improved cleanup, shutdown handling, and error reporting during execution failures. Documentation Clarified server reuse, shutdown behavior, interpreter discovery, and environment metadata.

Open Graph Description: Third in the chain: stacked on #519 (runtime hardening), which is stacked on #518 (interpreter resolution). Merge those first. Part of #162. Problem Every deepnote_run tool call in the MCP spawns a...

X Description: Third in the chain: stacked on #519 (runtime hardening), which is stacked on #518 (interpreter resolution). Merge those first. Part of #162. Problem Every deepnote_run tool call in the MCP spawns a...

Opengraph URL: https://github.com/deepnote/deepnote/pull/520

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width