Title: fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies by ericallam · Pull Request #4532 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies by ericallam · Pull Request #4532 · triggerdotdev/trigger.dev
X Title: fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies by ericallam · Pull Request #4532 · triggerdotdev/trigger.dev
Description: What The internal tracing ParentBasedSampler in tracer.server.ts left remoteParentSampled at its default of AlwaysOn. Any request arriving with a traceparent whose sampled flag was set got recorded in full, bypassing INTERNAL_OTEL_TRACE_SAMPLING_RATE entirely. Because the SDK propagates its (always-sampled) trace context on calls back to the platform from inside running tasks, the large majority of API server spans inherited a sampled parent and ignored the divisor. The sampling knob was effectively inert on the busiest service. This registers a custom propagator (NonInheritingTraceContextPropagator) that stops adopting the inbound trace as the parent: inject still delegates to the standard W3C trace + baggage propagators, so outbound propagation is unchanged. extract drops the parent span (trace.deleteSpan) while preserving baggage, so every incoming request roots its own trace and the ratio sampler applies uniformly. remoteParentSampled is also set to the ratio sampler as a belt-and-suspenders fallback, in case an inbound sampled parent ever reaches the sampler another way. Two effects: the divisor becomes effective on the API server, and the API no longer stitches onto (and inflates) the propagated task-run traces, which is where the very large, un-thinnable trace chains came from. Rooting each request removes those chains rather than only diluting them. Only the internal APM trace pipeline (INTERNAL_OTEL_TRACE_EXPORTER_URL) is affected. The user-facing run-trace pipeline (otel.v1.traces -> ClickHouse) is a separate path and is untouched. The only consumer of the global propagator's extract is the OTel HTTP/Express auto-instrumentation, so the blast radius is inbound-request trace shape. Evidence (local full-stack red/green, divisor 10) A local OTLP/JSON sink counting spans; a driver fires N requests at a real endpoint, each carrying a distinct sampled traceparent, then counts how many spans/traces carry that run's marker. run code sent kept traces kept fraction before unmodified 500 500 1.00 after this PR 500 67 0.134 after this PR 2000 213 0.1065 Before: 100% of inherited-sampled requests kept, divisor ignored. After: ~10% kept (the divisor), converging on it at larger N. In every after-run each kept request is a single self-rooted trace (kept spans == kept distinct traces), confirming the inherited chains are gone, not just thinned. typecheck passes. Rollout / rollback No flag. Behavior stays governed by the existing INTERNAL_OTEL_TRACE_SAMPLING_RATE. Rollback is a straight revert with no data migration. Notes Internal dashboards that count raw span or request volume from this pipeline will read lower once this ships. That is expected: those counts were inflated by the bypass, not a real drop in traffic. Latency/percentile monitors retain plenty of samples at the current divisor. refs TRI-13031
Open Graph Description: What The internal tracing ParentBasedSampler in tracer.server.ts left remoteParentSampled at its default of AlwaysOn. Any request arriving with a traceparent whose sampled flag was set got recorded...
X Description: What The internal tracing ParentBasedSampler in tracer.server.ts left remoteParentSampled at its default of AlwaysOn. Any request arriving with a traceparent whose sampled flag was set got recorded...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4532
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:1ca62f56-3525-0901-d503-db8556874b25 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | EE14:3552AC:346DF2E:4A0C9D6:6AFAF3C1 |
| html-safe-nonce | 20b307b62248d8fb331073cc340a5b3a5b49e146c6eeda1b972ed6ae5a024054 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFRTE0OjM1NTJBQzozNDZERjJFOjRBMEM5RDY6NkFGQUYzQzEiLCJ2aXNpdG9yX2lkIjoiNjc2MTcyMTgwNjUwODk3OTEzNyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 84a506e01a26cd3d7579bd5571c5cdc185cbedc865f3ec4771b4d6be85b9b9d5 |
| hovercard-subject-tag | pull_request:4228949780 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4532/files |
| twitter:image | https://avatars.githubusercontent.com/u/534?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/534?s=400&v=4 |
| og:image:alt | What The internal tracing ParentBasedSampler in tracer.server.ts left remoteParentSampled at its default of AlwaysOn. Any request arriving with a traceparent whose sampled flag was set got recorded... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | a85e96a83bed7871993e6b1ffda24105928eee17 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width