Title: perf(webapp,run-engine): cut CPU on the engine-facing worker-action routes by ericallam · Pull Request #4746 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: perf(webapp,run-engine): cut CPU on the engine-facing worker-action routes by ericallam · Pull Request #4746 · triggerdotdev/trigger.dev
X Title: perf(webapp,run-engine): cut CPU on the engine-facing worker-action routes by ericallam · Pull Request #4746 · triggerdotdev/trigger.dev
Description: Cuts CPU on the engine/v1/worker-actions/* routes a managed supervisor calls, and adds the benchmark harness the numbers come from. Measured on a local stack: on-CPU per completed run 9.07ms → 6.59ms (−27%), busy fraction 45.6% → 33.8%, with every worker-action p50 down 23–27%. Load was 5,000 runs / 24 virtual supervisors / 90s window / 30,120 requests / 0 errors. Query-count work from the same investigation is deliberately not here — it will follow as a separate PR. The three changes 1. Split the event-loop monitor in two (~14% of on-CPU, plus ~5pp of GC). eventLoopMonitor.server.ts installs a global async_hooks hook: init writes a Map entry for every async resource the process creates, before calls process.hrtime() and context.active() on every one. Enabling any async hook also puts V8 on the slow path for promise instrumentation process-wide. EVENT_LOOP_MONITOR_ENABLED defaulted to "1", so this was the shipping configuration. The blocked-loop detector is now opt-in (EVENT_LOOP_MONITOR_ENABLED, default 0). The event-loop utilization gauge — a single interval timer with no per-request cost — moves to its own flag (EVENT_LOOP_UTILIZATION_MONITOR_ENABLED, default 1) and stays on, so the useful half survives without the expensive half. A/B under identical load: monitor on monitor off change on-CPU per run 9.08ms 7.25ms −20% GC self time 9.80% 5.05% −4.75pp dequeue p50 76.6ms 62.8ms −18% attempts/start p50 56.3ms 43.5ms −23% 2. Bucket route matching by first static path segment (10.4% → 3.9% of on-CPU). patches/@remix-run__router@1.23.3.patch already memoized flattened branches and compiled path regexes. What remained was the linear scan: matchRouteBranch walked the ranked branch list calling matchPath per branch across 521 route files, so every worker-action request paid a scan proportional to the whole route table. Branches are now indexed by their lowercased leading segment, with one always-considered list for branches whose leading segment is dynamic, splat or optional (and for root/pathless paths). A request walks only its own bucket merged with that list. Route-matching self time dropped 64% (3.6s → 1.3s over a 90s window). Ordering is preserved exactly: both lists hold indexes into the already rank-sorted branch array and are walked in ascending-index order, so the first match found is the same branch the full scan would have found. Bucketing lowercases on both sides, so case-insensitive matching still resolves and caseSensitive: true routes are still rejected by matchPath itself. A pathname whose own leading segment can't be bucketed falls back to the full scan. Verified equivalent to the unpatched matcher over 20,050 pathnames (literal, dynamic, splat, optional, case variants, basenames, percent-encoded) with zero mismatches. apps/webapp/test/routeMatchingPatch.test.ts pins the matching semantics rather than the optimisation, so it still passes without the patch. 3. Demote per-heartbeat and per-dequeue info logs to debug. These are the two highest-rate engine calls and each wrote a synchronous structured log line on every request. Synchronous console writes can block the loop when stdout backs up, which costs more than the ~1.3% CPU share suggests. The harness Two benchmarks, neither in the default suite (they run for minutes, attach the V8 profiler, and report numbers rather than assert on them). See apps/webapp/test/bench/README.md. apps/webapp/test/bench/engineHttp.bench.test.ts — spawns a real webapp against throwaway Postgres/Redis containers, seeds a production environment with a promoted managed deployment, and drives a closed-loop supervisor pool through the full lifecycle. Profiling runs over CDP rather than --cpu-prof so it covers only the measured window instead of being swamped by boot, and performance.eventLoopUtilization() is sampled inside the webapp process. internal-packages/run-engine/src/engine/bench/runEngineLifecycle.bench.test.ts — drives RunEngine directly, profiling enqueue and lifecycle separately so engine cost isn't mixed with request-stack overhead. apps/webapp/test/bench/analyzeProfile.ts — dependency-free .cpuprofile analyzer that symbolicates through the build's source maps and ranks CPU by package, self time and total time. Percentages are shares of on-CPU time (V8's (idle)/(program) excluded). startWebapp gains overrideEnv, applied after the worker-disable defaults, so the HTTP bench can re-enable the run engine worker that drains the master queue into the worker queues a supervisor dequeues from. The local OTel collector gains a traces pipeline. It only defined a metrics pipeline, so pointing INTERNAL_OTEL_TRACE_EXPORTER_URL at it locally failed and the webapp silently fell back to the console span logger. Configuration For operators upgrading: EVENT_LOOP_MONITOR_ENABLED (now defaults to 0) — the per-async-resource blocked-loop detector. Set to 1 to restore the previous behaviour and keep emitting event-loop-blocked spans. EVENT_LOOP_UTILIZATION_MONITOR_ENABLED (new, defaults to 1) — the nodejs.event_loop.utilization gauge. Unchanged in behaviour; it just has its own flag now so it survives turning the detector off. Notes for review pnpm-lock.yaml changes only because the router patch content changed, which changes its patch hash. One thing the profile ruled out: with a real OTLP collector receiving spans, tracing costs ~1.7% of on-CPU at 100% sampling and ~0.8% at the production rate. Span shipping is not a hidden cost, so nothing here touches it. Caveats on the numbers: a laptop, not production hardware, so DB and Redis latency are unrepresentative (client-side CPU is what's ranked); single webapp process; throughput varies ~5% run to run, which is why the claims rest on on-CPU per run rather than req/s. Verification 20,050-pathname router equivalence check vs the unpatched matcher, zero mismatches apps/webapp/test/routeMatchingPatch.test.ts (12 cases) passes webapp e2e smoke suite (68 tests) passes through the patched router run-engine suites covering the snapshot/attempt paths pass typecheck, format, lint, knip clean
Open Graph Description: Cuts CPU on the engine/v1/worker-actions/* routes a managed supervisor calls, and adds the benchmark harness the numbers come from. Measured on a local stack: on-CPU per completed run 9.07ms → 6.59...
X Description: Cuts CPU on the engine/v1/worker-actions/* routes a managed supervisor calls, and adds the benchmark harness the numbers come from. Measured on a local stack: on-CPU per completed run 9.07ms → 6.59...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4746
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:ead3f177-afb6-f79d-3296-febb342f5e1a |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 90BA:22BBB3:4AD8:66CD:6AFB089B |
| html-safe-nonce | f8869a89d7e1687b98ac97080c863d94dc75e304cc4d382a7d5e587913e32b48 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MEJBOjIyQkJCMzo0QUQ4OjY2Q0Q6NkFGQjA4OUIiLCJ2aXNpdG9yX2lkIjoiNTA1NzMzMzE0NTQxMDE0NDQxMiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 711ccddcfe747888d3e1a1c5d2d5e8ac0fa74ce252b4b5defea919a52605611a |
| hovercard-subject-tag | pull_request:4329765472 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4746/files |
| twitter:image | https://avatars.githubusercontent.com/u/534?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/534?s=400&v=4 |
| og:image:alt | Cuts CPU on the engine/v1/worker-actions/* routes a managed supervisor calls, and adds the benchmark harness the numbers come from. Measured on a local stack: on-CPU per completed run 9.07ms → 6.59... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 76fb13cb43f1834708971f45a9cfcab5df3177a7582086613d7cdc6e98c4bfde |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | afc998826cdd963c8f91a96ae5372809ab5e296b |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width