Title: Self-hosted v4 runner pods leak process.env to stdout and to the webapp debug-log endpoint · Issue #3566 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: Self-hosted v4 runner pods leak process.env to stdout and to the webapp debug-log endpoint · Issue #3566 · triggerdotdev/trigger.dev
X Title: Self-hosted v4 runner pods leak process.env to stdout and to the webapp debug-log endpoint · Issue #3566 · triggerdotdev/trigger.dev
Description: Update: original report was incomplete. Two additional leak sites confirmed from production logs that do dump the full project env. See follow-up comment for the corrected scope and code references. Treat as security advisory. Summary On...
Open Graph Description: Update: original report was incomplete. Two additional leak sites confirmed from production logs that do dump the full project env. See follow-up comment for the corrected scope and code references...
X Description: Update: original report was incomplete. Two additional leak sites confirmed from production logs that do dump the full project env. See follow-up comment for the corrected scope and code references...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/issues/3566
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Self-hosted v4 runner pods leak process.env to stdout and to the webapp debug-log endpoint","articleBody":"> **Update:** original report was incomplete. Two additional leak sites confirmed from production logs that *do* dump the full project env. See follow-up comment for the corrected scope and code references. Treat as security advisory.\n\n## Summary\n\nOn every runner pod startup, the managed run controller emits **three** distinct log entries whose payloads include sensitive material:\n\n1. **\"Creating run controller\"** (`packages/cli-v3/src/entryPoints/managed/controller.ts:84`) — spreads `...env.raw` (parsed `RunnerEnv` schema) into the log properties.\n2. **\"started attempt\"** (`packages/cli-v3/src/entryPoints/managed/execution.ts:448`) — logs `{ start: start.data }`, where `start.data` is the `startRunAttempt` API response **including the project `envVars` map** (all user-provided env vars passed to the task — API keys, DB creds, JWT signing certs, etc.).\n3. **\"initializing task run process\"** (`packages/cli-v3/src/executions/taskRunProcess.ts:152`) — logs `{ env: fullEnv, ... }` where `fullEnv = { ...$env, NODE_OPTIONS, PATH, ... }` and `$env` is the same project env from the IPC message. Same secrets dumped a second time.\n\nAll three entries go to the runner pod's stdout (operator-side log aggregation: CloudWatch / Loki / Datadog / …). #1 and #2 additionally go to the webapp's debug-log HTTP endpoint, which the webapp persists and renders in the dashboard run timeline.\n\nSetting `DEBUG=false`, `LOG_LEVEL=info`, `VERBOSE=false`, or `PRETTY_LOGS=false` does **not** silence any of these. `managed-run-controller.ts:7` hardcodes `logger.loggerLevel = \"debug\"` so site #3 always fires regardless of operator config. Sites #1 and #2 go through `ManagedRunLogger.sendDebugLog`, whose local sink uses `SimpleStructuredLogger.log` (i.e. `LogLevel.log = 0` — lowest in the enum; the guard `if (this.level \u003c LogLevel.log) return;` cannot suppress it).\n\nConfirmed on chart `4.4.5` / app image `v4.4.5`. The code paths exist on `main` at the time of writing.\n\n## Impact\n\nFor any self-hosted v4 installation that runs tasks needing any kind of API key (LLM provider, error tracker, third-party APIs, database, JWT signing, etc.), the full set of those secrets is written:\n\n- To operator-side log storage on every runner pod startup.\n- To the webapp's debug-log endpoint, which persists them and shows them in the dashboard run timeline (site #2).\n\nIn my own production-gov logs, I can see real values for Auth0 client secret + management API client secret + signing secret, Hasura admin secret, Hasura JWT RS512 signing certificate, AWS access key + secret, OpenAI / Sendgrid / Datadog / Sentry / Apollo / Merge / Kernel / Cerebras / OpenExchangeRates API keys, the Trigger DB password embedded in `TRIGGER_DATABASE_URL`, and per-run `TRIGGER_JWT`.\n\n## Source\n\n### Site #1 — `packages/cli-v3/src/entryPoints/managed/controller.ts:77-87`\n\n```ts\nconst properties = {\n ...env.raw, // parsed RunnerEnv — bounded by schema\n TRIGGER_POD_SCHEDULED_AT_MS: env.TRIGGER_POD_SCHEDULED_AT_MS.toISOString(),\n TRIGGER_DEQUEUED_AT_MS: env.TRIGGER_DEQUEUED_AT_MS.toISOString(),\n};\n\nthis.sendDebugLog({\n runId: env.TRIGGER_RUN_ID,\n message: \"Creating run controller\",\n properties,\n});\n```\n\n### Site #2 — `packages/cli-v3/src/entryPoints/managed/execution.ts:448`\n\n```ts\nthis.sendDebugLog(\"started attempt\", { start: start.data });\n```\n\n`start.data` is the body of `httpClient.startRunAttempt()`, which includes `envVars` — the project env map passed to the task. Concretely: every env var configured on the Trigger project.\n\n### Site #3 — `packages/cli-v3/src/executions/taskRunProcess.ts:152`\n\n```ts\nconst fullEnv = {\n ...$env,\n OTEL_IMPORT_HOOK_INCLUDES: ...,\n NODE_OPTIONS: ...,\n PATH: process.env.PATH,\n TRIGGER_PROCESS_FORK_START_TIME: ...,\n TRIGGER_WARM_START: ...,\n TRIGGERDOTDEV: \"1\",\n};\n\nlogger.debug(\\`initializing task run process\\`, {\n env: fullEnv,\n path: workerManifest.workerEntryPoint,\n cwd,\n});\n```\n\n`$env` is the project env passed in from the IPC `EXECUTE_TASK_RUN` message — same secret material as site #2.\n\n### `ManagedRunLogger.sendDebugLog` (the routing for sites #1 and #2)\n\n`packages/cli-v3/src/entryPoints/managed/logger.ts:38-66`:\n\n```ts\nif (print) {\n this.logger.log(message, mergedProperties); // stdout via SimpleStructuredLogger\n}\nthis.httpClient.sendDebugLog(runId, { // POSTed to webapp\n message,\n time: date ?? new Date(),\n properties: flattenedProperties,\n});\n```\n\nThe webapp endpoint persists the properties and surfaces them in the dashboard run timeline.\n\n## Suggested fix\n\nThree changes, one per site:\n\n1. **`controller.ts`** — drop `...env.raw` from the `properties` spread, or replace with an explicit allowlist (e.g. the supervisor's `RunnerEnv` schema is a fine starting point if you want to keep some diagnostics).\n2. **`execution.ts:448`** — log a redacted summary of `start.data` (attempt number, snapshot id, run id, machine preset) rather than the raw API response body. At minimum, strip `envVars` before logging.\n3. **`taskRunProcess.ts:152`** — drop the `env: fullEnv` field from the log entry, or redact (e.g. log only the *keys* of `fullEnv`, never values).\n\nBonus: the `ManagedRunLogger.sendDebugLog` HTTP sink ignores the `print` flag and always forwards to the webapp endpoint. Worth considering whether `print: false` (or a new `persist: false`) should also gate the HTTP sink, so operators have a way to disable persistence of incidentally-sensitive properties without losing local diagnostics.\n\nHappy to open a PR. Filing this as an issue first so you can decide on scope.","author":{"url":"https://github.com/brentshulman-silkline","@type":"Person","name":"brentshulman-silkline"},"datePublished":"2026-05-12T18:18:40Z","url":"https://github.com/triggerdotdev/trigger.dev/issues/3566"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:c098fe2b-b1a3-3f90-622f-35a4439de25a |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 9AA6:1F63:80732D:C33139:6AFAE9D1 |
| html-safe-nonce | c6be8faa81f1ce1c5272ae333d699d60bc0e575e85187f64919c21673c64776d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5QUE2OjFGNjM6ODA3MzJEOkMzMzEzOTo2QUZBRTlEMSIsInZpc2l0b3JfaWQiOiIxODAzNTQ4MzU0MjQxNDM4MjUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 119c9a371ab153065d7659bdfdc5cb3cb798c7e0619dc2f064efa00b570c307d |
| hovercard-subject-tag | issue:4431567165 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/triggerdotdev/trigger.dev/3566/issue_layout |
| twitter:image | https://opengraph.githubassets.com/3360878b50fa7e93b0945127e84e14296bf1337094171c97de36995e10498f27/triggerdotdev/trigger.dev/issues/3566 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/3360878b50fa7e93b0945127e84e14296bf1337094171c97de36995e10498f27/triggerdotdev/trigger.dev/issues/3566 |
| og:image:alt | Update: original report was incomplete. Two additional leak sites confirmed from production logs that do dump the full project env. See follow-up comment for the corrected scope and code references... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | brentshulman-silkline |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 12fff95459b24b0f882a66698ee6a36a5934d247 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width