René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Memory exhaustion in loop executions with agent blocks making tool calls","articleBody":"## Summary\n\nWhen running workflows with loops containing agent blocks that make many tool calls (e.g., MCP file operations), memory accumulates unbounded until system OOM. This caused my 64GB system to run out of memory during workflow execution.\n\n## Environment\n\n- Self-hosted via Docker\n- macOS with 64GB RAM\n- Docker containers with no memory limits (default)\n\n## Root Cause Analysis\n\nAfter investigating the codebase, I identified **6 critical memory accumulation points** with no bounds:\n\n### 1. `allIterationOutputs` in LoopScope (CRITICAL)\n\n**File:** `apps/sim/executor/orchestrators/loop.ts` (line 144)\n\n```typescript\nif (iterationResults.length \u003e 0) {\n  scope.allIterationOutputs.push(iterationResults)  // UNBOUNDED\n}\n```\n\nEvery loop iteration pushes results to this array with no size limit or pruning.\n\n### 2. `blockLogs` array in ExecutionContext (CRITICAL)\n\n**File:** `apps/sim/executor/execution/block-executor.ts` (lines 60-62)\n\n```typescript\nblockLog = this.createBlockLog(ctx, node.id, block, node)\nctx.blockLogs.push(blockLog)  // UNBOUNDED - every block execution adds\n```\n\nEvery block execution (including each loop iteration) pushes to this array.\n\n### 3. TraceSpans accumulation\n\n**File:** `apps/sim/lib/logs/execution/trace-spans/trace-spans.ts`\n\nThe `buildTraceSpans()` function creates spans for ALL block logs and tool calls, building an unbounded in-memory tree structure.\n\n### 4. Tool calls array per agent\n\n**File:** `apps/sim/executor/handlers/agent/agent-handler.ts` (lines 250-318)\n\nEach agent execution stores all tool calls in `span.toolCalls` array. With 100 tool calls per agent × 1000 iterations = 100,000 ToolCall objects.\n\n### 5. File extraction unbounded recursion\n\n**File:** `apps/sim/lib/logs/execution/logger.ts` (lines 621-695)\n\n`extractFilesFromExecution()` recursively walks the entire trace span tree with no depth limits.\n\n### 6. Memory table JSONB concatenation\n\n**File:** `apps/sim/executor/handlers/agent/memory.ts` (line ~580)\n\n```typescript\ndata: sql`${memory.data} || ${JSON.stringify([message])}::jsonb`  // Unbounded append\n```\n\n## Memory Growth Calculation\n\nFor a workflow with:\n- Loop: 1000 iterations\n- Agent block per iteration with 100 tool calls (file read/write via MCP)\n\n**Memory consumption:**\n- `blockLogs`: 1000 entries × 50KB = 50MB\n- `allIterationOutputs`: 1000 × 1MB = 1GB  \n- Tool calls in spans: 100,000 objects = 1GB\n- TraceSpans tree traversal: O(GB) object graph\n- **Total**: 2-10+ GB in memory, persisted to `executionData` JSONB\n\n## Reproduction Steps\n\n1. Create a workflow with a Loop block (while condition)\n2. Inside loop: StateTracker function → Agent with MCP tools (file read/write) → Reviewer\n3. Run the workflow with multiple tasks that require many file operations\n4. Monitor Docker container memory with `docker stats`\n5. Observe memory climbing until OOM\n\n## Suggested Fixes\n\n### Short-term mitigations:\n\n1. **Add iteration limits check with memory awareness:**\n```typescript\nconst MAX_MEMORY_PER_LOOP = 100 * 1024 * 1024; // 100MB\nif (estimateSize(scope.allIterationOutputs) \u003e MAX_MEMORY_PER_LOOP) {\n  logger.warn('Loop memory limit reached, truncating older iterations');\n  scope.allIterationOutputs = scope.allIterationOutputs.slice(-10);\n}\n```\n\n2. **Stream block logs to database instead of accumulating:**\n```typescript\n// Instead of ctx.blockLogs.push(blockLog)\nawait this.persistBlockLog(blockLog);\n```\n\n3. **Truncate tool call results in spans:**\n```typescript\nconst MAX_TOOL_RESULT_SIZE = 10000;\ntoolCall.result = truncate(toolCall.result, MAX_TOOL_RESULT_SIZE);\n```\n\n### Long-term solutions:\n\n1. Implement streaming execution logs (write to DB incrementally)\n2. Add configurable memory limits per execution\n3. Implement log rotation/pruning during long-running executions\n4. Add memory pressure monitoring with graceful degradation\n\n## Workaround\n\nSet Docker memory limits to prevent system-wide OOM:\n\n```yaml\n# docker-compose.override.yml\nservices:\n  simstudio-app:\n    deploy:\n      resources:\n        limits:\n          memory: 8G\n```\n\n## Related Files\n\n- `apps/sim/executor/orchestrators/loop.ts`\n- `apps/sim/executor/execution/block-executor.ts`\n- `apps/sim/lib/logs/execution/trace-spans/trace-spans.ts`\n- `apps/sim/lib/logs/execution/logger.ts`\n- `apps/sim/executor/handlers/agent/agent-handler.ts`\n- `apps/sim/executor/handlers/agent/memory.ts`\n- `packages/db/schema.ts` (executionData JSONB field)","author":{"url":"https://github.com/aadamsx","@type":"Person","name":"aadamsx"},"datePublished":"2025-12-22T18:38:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/2525/sim/issues/2525"}