Title: `perf(supervisor): sequential inspect loop in DockerResourceMonitor blocks task scheduling thread` · Issue #3727 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: `perf(supervisor): sequential inspect loop in DockerResourceMonitor blocks task scheduling thread` · Issue #3727 · triggerdotdev/trigger.dev
X Title: `perf(supervisor): sequential inspect loop in DockerResourceMonitor blocks task scheduling thread` · Issue #3727 · triggerdotdev/trigger.dev
Description: Bug Description The DockerResourceMonitor.getNodeResources() method calculates system resource consumption by iterating through all containers. During each iteration, it performs a sequential inspect call to retrieve CPU and memory limit...
Open Graph Description: Bug Description The DockerResourceMonitor.getNodeResources() method calculates system resource consumption by iterating through all containers. During each iteration, it performs a sequential inspe...
X Description: Bug Description The DockerResourceMonitor.getNodeResources() method calculates system resource consumption by iterating through all containers. During each iteration, it performs a sequential inspe...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/issues/3727
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`perf(supervisor): sequential inspect loop in DockerResourceMonitor blocks task scheduling thread`","articleBody":"### Bug Description\nThe `DockerResourceMonitor.getNodeResources()` method calculates system resource consumption by iterating through all containers. During each iteration, it performs a sequential inspect call to retrieve CPU and memory limits:\n\n```typescript\nfor (const container of stats) {\n if (container.State === \"running\") {\n const c = this.docker.getContainer(container.Id);\n const { HostConfig } = await c.inspect(); // Sequential blocking call\n\n const cpu = this.resourceParser.cpu(HostConfig.NanoCpus ?? 0);\n const memory = this.resourceParser.memory(HostConfig.Memory ?? 0);\n\n cpuUsed += cpu;\n memoryUsed += memory;\n }\n}\n```\n\nBecause `getNodeResources` is called frequently to make container placement decisions during scheduling, this sequential inspect loop acts as a synchronous bottleneck. On docker hosts running many containers, performing sequential inspect round-trips to the Unix socket blocks the scheduling thread for seconds and creates workload placement latency.\n\n### Root Cause File\n`apps/supervisor/src/resourceMonitor.ts`\n\n### Steps to Reproduce\n1. Start the Trigger.dev supervisor daemon on a machine hosting 50+ active docker containers.\n2. Trigger multiple rapid task scheduling executions (`wouldFit`).\n3. Notice scheduling latency and elevated CPU thread blocking inside the supervisor daemon due to sequential docker socket round-trips.\n\n### Proposed Fix\nUse `Promise.all` to fetch container configurations concurrently rather than sequentially:\n\n```typescript\nconst runningContainers = stats.filter(container => container.State === \"running\");\nconst inspections = await Promise.all(\n runningContainers.map(async (container) => {\n const c = this.docker.getContainer(container.Id);\n const { HostConfig } = await c.inspect();\n return {\n cpu: this.resourceParser.cpu(HostConfig.NanoCpus ?? 0),\n memory: this.resourceParser.memory(HostConfig.Memory ?? 0),\n };\n })\n);\n\nfor (const inspect of inspections) {\n cpuUsed += inspect.cpu;\n memoryUsed += inspect.memory;\n}\n```","author":{"url":"https://github.com/KanhaiyaBagul","@type":"Person","name":"KanhaiyaBagul"},"datePublished":"2026-05-23T11:10:44Z","url":"https://github.com/triggerdotdev/trigger.dev/issues/3727"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:1c116400-0561-df73-931f-4de57c69d072 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8C7E:3FAA02:594E9C:8414EF:6AFAEEE4 |
| html-safe-nonce | a4a6f5c5079148a2c992e14b2d36d387025c6057ac94f680c3f8adeb51e8e048 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4QzdFOjNGQUEwMjo1OTRFOUM6ODQxNEVGOjZBRkFFRUU0IiwidmlzaXRvcl9pZCI6IjgxNzc1ODU4NTYxMzE1NTkxNDAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | e222866071946e0b22599ba24f1117cd7f3debb00f277834452fea0fe1a4449b |
| hovercard-subject-tag | issue:4507871876 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/triggerdotdev/trigger.dev/3727/issue_layout |
| twitter:image | https://opengraph.githubassets.com/c19cdf8f3b929678e2671e429d0270ca1dffdb99290a6a55eadc27bf7827beee/triggerdotdev/trigger.dev/issues/3727 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/c19cdf8f3b929678e2671e429d0270ca1dffdb99290a6a55eadc27bf7827beee/triggerdotdev/trigger.dev/issues/3727 |
| og:image:alt | Bug Description The DockerResourceMonitor.getNodeResources() method calculates system resource consumption by iterating through all containers. During each iteration, it performs a sequential inspe... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | KanhaiyaBagul |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 12fff95459b24b0f882a66698ee6a36a5934d247 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width