Title: bug(v4): TracingSDK.flush() uses Promise.all, drops trace + log exports on shutdown when one provider rejects · Issue #3556 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: bug(v4): TracingSDK.flush() uses Promise.all, drops trace + log exports on shutdown when one provider rejects · Issue #3556 · triggerdotdev/trigger.dev
X Title: bug(v4): TracingSDK.flush() uses Promise.all, drops trace + log exports on shutdown when one provider rejects · Issue #3556 · triggerdotdev/trigger.dev
Description: Problem On self-hosted Trigger.dev v4.4.5, the user task's logger.info(...) calls and child-trigger spans are silently dropped from a run's dashboard trace when processKeepAliveEnabled is false (the default for self-host). Engine state e...
Open Graph Description: Problem On self-hosted Trigger.dev v4.4.5, the user task's logger.info(...) calls and child-trigger spans are silently dropped from a run's dashboard trace when processKeepAliveEnabled is false (th...
X Description: Problem On self-hosted Trigger.dev v4.4.5, the user task's logger.info(...) calls and child-trigger spans are silently dropped from a run's dashboard trace when processKeepAliveEnabled is f...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/issues/3556
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"bug(v4): TracingSDK.flush() uses Promise.all, drops trace + log exports on shutdown when one provider rejects","articleBody":"## Problem\n\n\u003cimg width=\"1695\" height=\"655\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/c6513711-0fd5-418c-bd91-9b66db18d855\" />\n\n\nOn self-hosted Trigger.dev v4.4.5, the user task's `logger.info(...)` calls and child-trigger spans are silently dropped from a run's dashboard trace when `processKeepAliveEnabled` is `false` (the default for self-host). Engine state events (`QUEUED` / `PENDING_EXECUTING` / `EXECUTING` / `run:subscribe`) still appear because they're written server-side; only the user-emitted telemetry vanishes.\n\n## Environment\n\n- Trigger.dev v4.4.5 (Helm chart `oci://ghcr.io/triggerdotdev/charts/trigger:4.4.5`, image `ghcr.io/triggerdotdev/trigger.dev:v4.4.5`)\n- Self-hosted on Kubernetes (EKS), single-shard ClickHouse from the chart\n- Node 22.16.0 task runtime, `processKeepAliveEnabled: false`\n- `OTEL_EXPORTER_OTLP_ENDPOINT=http://trigger-webapp:3030/otel`\n\n## Root cause\n\n`TracingSDK.flush()` at [`packages/core/src/v3/otel/tracingSDK.ts#L371-L377`](https://github.com/triggerdotdev/trigger.dev/blob/v4.4.5/packages/core/src/v3/otel/tracingSDK.ts#L371-L377):\n\n```ts\npublic async flush() {\n await Promise.all([\n this._traceProvider.forceFlush(),\n this._logProvider.forceFlush(),\n this._meterProvider.forceFlush(),\n ]);\n}\n```\n\n`Promise.all` semantics: the first rejection rejects the combined promise immediately, abandoning the other two providers' in-flight HTTP exports. The runner-side flush is invoked from the worker handler at [`packages/cli-v3/src/entryPoints/managed-run-worker.ts#L716-L731`](https://github.com/triggerdotdev/trigger.dev/blob/v4.4.5/packages/cli-v3/src/entryPoints/managed-run-worker.ts#L716-L731), which races a 5 s timeout — but a rejection short-circuits the race in milliseconds.\n\n`TracingSDK.shutdown()` at L379-L385 has the same bug.\n\n## Evidence\n\nRunner pod stdout at task completion (from CloudWatch, redacted run ID):\n\n```\n00:25:54.571 [execution] executing task run process ← task fork spawned\n ── 13.2 s of task code (with multiple logger.info calls + batchTrigger spans) ──\n00:26:07.791 [execution] completed run attempt\n00:26:07.792 [taskRunProcessProvider] Keep-alive disabled, cleaning up process immediately\n00:26:07.795 Flushed usage in 1.032 ms\n00:26:07.797 Flushed runMetadata successfully\n00:26:07.797 Failed to flush tracingSDK ← here\n00:26:07.797 Flushed all in 4.039 ms\n00:26:07.878 [controller] Exiting process\n```\n\nTotal tracingSDK flush attempt: **4 ms** — well under the 5000 ms timeout. So one of `_traceProvider.forceFlush()` / `_logProvider.forceFlush()` / `_meterProvider.forceFlush()` rejected effectively synchronously, taking the entire flush down with it via \\`Promise.all\\`.\n\nWe verified separately that the backing infra was fully healthy throughout the window:\n- ClickHouse pod: 0 restarts, ~34% memory utilization, container Running, PVC bound.\n- Webapp's `/otel/v1/logs` endpoint was returning steady 200s.\n- Webapp's `ClickhouseEventRepository.flushBatch` was logging `Inserted batch into clickhouse` with non-zero `written_rows` every few seconds.\n\nSo the data simply never leaves the runner.\n\nMost likely rejecter: `_meterProvider.forceFlush()`. The metrics pipeline is the newest moving part in this path — wraps the OTLP metric exporter in a custom `BufferingMetricExporter` (tracingSDK.ts:311), introduced in #3061 with follow-up #3106. The trace and log paths are older and use stock OTel batch processors. But the `Promise.all` bug applies regardless of which provider rejects.\n\n## Suggested fix\n\nSwap `Promise.all` for `Promise.allSettled` in both `TracingSDK.flush()` and `TracingSDK.shutdown()`, and log per-provider rejections without bubbling them up:\n\n```ts\npublic async flush() {\n const results = await Promise.allSettled([\n this._traceProvider.forceFlush(),\n this._logProvider.forceFlush(),\n this._meterProvider.forceFlush(),\n ]);\n results.forEach((r, i) => {\n if (r.status === 'rejected') {\n const name = ['trace', 'log', 'meter'][i];\n console.error(\\`Failed to flush \\${name} provider:\\`, r.reason);\n }\n });\n}\n```\n\nThat preserves visibility into which provider failed (currently lost—the worker just logs \"Failed to flush tracingSDK\" with no provider attribution) while letting the other two providers' flushes complete rather than being abandoned.\n\n## Impact\n\nAny task running on a worker with `processKeepAliveEnabled: false` is at risk of losing all user-emitted trace data on shutdown if any of the three providers rejects `forceFlush()`. For short tasks (under ~5–10 s), this manifests as a run with engine events only and no user spans/logs.\n\n## Reproduction (qualitative)\n\n1. Self-host v4.4.5 with the default Helm values.\n2. Deploy a task that calls \\`logger.info(...)\\` a few times and does \\`tasks.batchTriggerAndWait\\` on a small batch. Keep total task duration under 30 s.\n3. Trigger the task in prod.\n4. Open the run trace in the dashboard 1 min later.\n5. Observe: engine events present; \\`logger.info\\` lines and \\`batchTriggerAndWait\\` spans absent. Runner pod stdout (if accessible) shows \\`Failed to flush tracingSDK\\`.","author":{"url":"https://github.com/brentshulman-silkline","@type":"Person","name":"brentshulman-silkline"},"datePublished":"2026-05-12T01:17:02Z","url":"https://github.com/triggerdotdev/trigger.dev/issues/3556"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:867d4952-f2bd-c9d7-97c4-1386ccd3ac3c |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | CD5A:1C77AB:30F997C:457B677:6AFAE9C9 |
| html-safe-nonce | 5735fb04c49af57b98c55e8a4196ee2572044fe8725d36027cfb580aa4c03d32 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRDVBOjFDNzdBQjozMEY5OTdDOjQ1N0I2Nzc6NkFGQUU5QzkiLCJ2aXNpdG9yX2lkIjoiNjExMzI3MTAwNzExODI4OTM1MyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 404e0179bec20bf6a709b10afdcbcf553b16e53a3cc58826ecde58eb182e38b7 |
| hovercard-subject-tag | issue:4425440661 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/triggerdotdev/trigger.dev/3556/issue_layout |
| twitter:image | https://opengraph.githubassets.com/e2f3f604a3fdb5379666095efe4ab1128fbab4532796b3798924ecc01a70b135/triggerdotdev/trigger.dev/issues/3556 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/e2f3f604a3fdb5379666095efe4ab1128fbab4532796b3798924ecc01a70b135/triggerdotdev/trigger.dev/issues/3556 |
| og:image:alt | Problem On self-hosted Trigger.dev v4.4.5, the user task's logger.info(...) calls and child-trigger spans are silently dropped from a run's dashboard trace when processKeepAliveEnabled is false (th... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | brentshulman-silkline |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 12fff95459b24b0f882a66698ee6a36a5934d247 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width