Title: fix(core): mint the fallback external trace id per run by NERLOE · Pull Request #4526 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: fix(core): mint the fallback external trace id per run by NERLOE · Pull Request #4526 · triggerdotdev/trigger.dev
X Title: fix(core): mint the fallback external trace id per run by NERLOE · Pull Request #4526 · triggerdotdev/trigger.dev
Description: Problem Runs that carry no external trace context (schedules, task-to-task triggers, anything not started from an incoming traceparent) fall back to a generated external trace id. That id is generated once, in the TracingSDK constructor: https://github.com/triggerdotdev/trigger.dev/blob/main/packages/core/src/v3/otel/tracingSDK.ts#L165 With experimental_processKeepAlive enabled, the TracingSDK outlives the run, so every run that executes on a warm process is exported to the external OTLP endpoint under that same trace id and unrelated runs get merged into one trace on the receiving backend. This is the same warm-start hazard that c043c4a fixed for the external-context path. That commit made the wrappers read traceContext.getExternalTraceContext() live instead of capturing it at construction, but deliberately left the fallback captured, so the bug survives for exactly the runs that have no external context. What it looks like in production We export to a self-hosted Langfuse via telemetry.exporters. Measured over our production traces: 80.3% of traces contain spans from more than one Trigger run worst case: 25 distinct runs collapsed into a single trace Per-trace cost and latency attribution is meaningless as a result: a trace shows an unrelated mix of workloads, and drilling into one run is impossible. Disabling experimental_processKeepAlive avoids it, but that is a significant throughput regression and not a real option for us. Fix FallbackExternalTraceId holds the generated id and remints it when the run changes. The TracingSDK constructs one instance and passes it to every ExternalSpanExporterWrapper and ExternalLogRecordExporterWrapper, so a run's spans and logs agree on the id after a remint. That matches the old behaviour, where all wrappers received one identical string. The run boundary comes from the manager rather than being inferred. StandardTraceContextManager.traceContext becomes an accessor pair that advances an epoch whenever the context is replaced, which is exactly what starting a run does, so no call site changes. getTraceContextEpoch() is added to TraceContextManager, and the noop manager reports a constant, so with no manager registered there are no boundaries to react to. I did first try inferring the boundary from reference-identity of getTraceContext(). It works, but guarding it against the noop manager's freshly allocated {} requires testing the context for emptiness, and since traceContext is z.record(z.unknown()) that silently stops reminting for a run whose context is legitimately empty, merging it back into the previous run's trace. The epoch avoids the heuristic entirely. One behaviour held deliberately: an empty configured id still means external export is off, so the empty seed short-circuits rather than minting an id and switching the feature on for a deployment that never asked for it. Tests packages/core/test/externalSpanExporterWrapper.test.ts gains six cases: mints a new fallback trace id per run when there is no external context mints a new fallback trace id for a run whose trace context is empty keeps one fallback trace id across every export within a run leaves external export off when no external trace id was configured keeps a run's spans and logs on the same id after a remint holds the id when no trace context manager is registered Mutation-checked. Removing the epoch bump fails three of them, and reinstating the emptiness heuristic fails the empty-context case with the exact symptom it describes. Full packages/core suite passes (674 tests). The harness needed one fix to make these meaningful. traceContext.setGlobalManager() delegates to registerGlobal, which ignores a second registration, so the beforeEach only ever installed the first test's manager and every later test was mutating an object that was no longer global. Calling traceContext.disable() first makes each test's manager actually take effect. Happy to split the traceContext epoch into its own commit or PR if you'd rather review it separately, or to take a different approach to the boundary entirely. 🤖 Generated with Claude Code
Open Graph Description: Problem Runs that carry no external trace context (schedules, task-to-task triggers, anything not started from an incoming traceparent) fall back to a generated external trace id. That id is genera...
X Description: Problem Runs that carry no external trace context (schedules, task-to-task triggers, anything not started from an incoming traceparent) fall back to a generated external trace id. That id is genera...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4526
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:fcadf5ce-3efc-73bf-c27f-ea32edd8b8d0 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | A7FC:1E77:5FEFCD:8F5A87:6AFAF20E |
| html-safe-nonce | c5e5f1978ba737d8b5c5affdd88d15654b9a39dbb0f6c5fb97bf4d1ebb62e0c5 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBN0ZDOjFFNzc6NUZFRkNEOjhGNUE4Nzo2QUZBRjIwRSIsInZpc2l0b3JfaWQiOiIzMjIzNjUwMzkwODU1MzE1OTgyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 101097d979644fb62dbbe0d01481cde1d1787938875be43e4c9eca9e40b52f73 |
| hovercard-subject-tag | pull_request:4227046380 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4526/files |
| twitter:image | https://avatars.githubusercontent.com/u/32075361?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/32075361?s=400&v=4 |
| og:image:alt | Problem Runs that carry no external trace context (schedules, task-to-task triggers, anything not started from an incoming traceparent) fall back to a generated external trace id. That id is genera... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | a85e96a83bed7871993e6b1ffda24105928eee17 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width