Title: Kubernetes: run telemetry has empty host — TRIGGER_WORKER_INSTANCE_NAME (spec.nodeName) not forwarded into the run process · Issue #4214 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: Kubernetes: run telemetry has empty host — TRIGGER_WORKER_INSTANCE_NAME (spec.nodeName) not forwarded into the run process · Issue #4214 · triggerdotdev/trigger.dev
X Title: Kubernetes: run telemetry has empty host — TRIGGER_WORKER_INSTANCE_NAME (spec.nodeName) not forwarded into the run process · Issue #4214 · triggerdotdev/trigger.dev
Description: Summary On self-hosted Kubernetes, task-run spans and logs exported over OTLP to an off-node collector (e.g. a centralized OpenTelemetry/Datadog Agent) arrive with no host. Datadog tags every such span issue_type:empty_hostname. There is...
Open Graph Description: Summary On self-hosted Kubernetes, task-run spans and logs exported over OTLP to an off-node collector (e.g. a centralized OpenTelemetry/Datadog Agent) arrive with no host. Datadog tags every such ...
X Description: Summary On self-hosted Kubernetes, task-run spans and logs exported over OTLP to an off-node collector (e.g. a centralized OpenTelemetry/Datadog Agent) arrive with no host. Datadog tags every such ...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/issues/4214
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Kubernetes: run telemetry has empty host — TRIGGER_WORKER_INSTANCE_NAME (spec.nodeName) not forwarded into the run process","articleBody":"## Summary\n\nOn self-hosted **Kubernetes**, task-run spans and logs exported over OTLP to an **off-node** collector (e.g. a centralized OpenTelemetry/Datadog Agent) arrive with **no host**. Datadog tags every such span `issue_type:empty_hostname`. There is currently no supported way for a user's `trigger.config.ts` telemetry `resource` to recover the node, because the node identity never reaches the run process — even though the supervisor already puts it on the pod.\n\n## Root cause\n\nThe Kubernetes workload manager already injects the node name into the **run-controller container** via the downward API — `apps/supervisor/src/workloadManager/kubernetes.ts`:\n\n```ts\n{\n name: \"TRIGGER_WORKER_INSTANCE_NAME\",\n valueFrom: { fieldRef: { fieldPath: \"spec.nodeName\" } },\n},\n```\n\nBut the actual task-run worker is `fork()`ed with an **explicit, replaced env** (not the container's `process.env`):\n\n- `packages/cli-v3/src/entryPoints/managed/taskRunProcessProvider.ts` → `buildProcessEnvironment()` builds the child env as `{ ...taskRunEnv, ...this.env.gatherProcessEnv(), TRIGGER_MACHINE_ID, HEARTBEAT_INTERVAL_MS }`.\n- `packages/cli-v3/src/executions/taskRunProcess.ts` → `fork(..., { env: fullEnv })` where `fullEnv = { ...$env, OTEL_IMPORT_HOOK_INCLUDES, NODE_OPTIONS, PATH, TRIGGER_PROCESS_FORK_START_TIME, TRIGGER_WARM_START, TRIGGERDOTDEV }`.\n- `packages/cli-v3/src/entryPoints/managed/env.ts` → `RunnerEnv.gatherProcessEnv()` returns a **fixed allowlist**: `NODE_ENV, NODE_EXTRA_CA_CERTS, OTEL_EXPORTER_OTLP_ENDPOINT, TRIGGER_OTEL_EXPORTER_OTLP_ENDPOINT, UV_USE_IO_URING`.\n\nSo in the run process, `process.env.TRIGGER_WORKER_INSTANCE_NAME` (and `HOSTNAME`) are **undefined**. The node name exists on the pod but is dropped before it can be attached to the OTel resource built from `config.telemetry.resource`.\n\nFor comparison, the **Docker** workload manager already forwards it to the runner (`apps/supervisor/src/workloadManager/docker.ts`): `` `TRIGGER_WORKER_INSTANCE_NAME=${env.TRIGGER_WORKER_INSTANCE_NAME}` `` — so k8s is the inconsistent path here.\n\n## Requested change (one line)\n\nForward `TRIGGER_WORKER_INSTANCE_NAME` from the run process env allowlist. It is already in the `Env` schema and has a getter (`get TRIGGER_WORKER_INSTANCE_NAME()`), so this is additive and safe (the existing `undefined` filter drops it when unset).\n\n**File:** `packages/cli-v3/src/entryPoints/managed/env.ts`, method `RunnerEnv.gatherProcessEnv()`\n\n```diff\n const $env = {\n NODE_ENV: this.NODE_ENV,\n NODE_EXTRA_CA_CERTS: this.NODE_EXTRA_CA_CERTS,\n OTEL_EXPORTER_OTLP_ENDPOINT: this.OTEL_EXPORTER_OTLP_ENDPOINT,\n TRIGGER_OTEL_EXPORTER_OTLP_ENDPOINT: this.OTEL_EXPORTER_OTLP_ENDPOINT,\n UV_USE_IO_URING: this.UV_USE_IO_URING,\n+ // Node the run is executing on (spec.nodeName on k8s). Forwarding it lets\n+ // trigger.config.ts telemetry set an OTel host.name so run traces/logs\n+ // exported to an off-node collector are attributed to a host.\n+ TRIGGER_WORKER_INSTANCE_NAME: this.TRIGGER_WORKER_INSTANCE_NAME,\n };\n```\n\nWith that, a user can map it in `trigger.config.ts`:\n\n```ts\nconst nodeName = process.env.TRIGGER_WORKER_INSTANCE_NAME; // = spec.nodeName on k8s\nexport default defineConfig({\n telemetry: {\n resource: resourceFromAttributes(\n nodeName ? { \"host.name\": nodeName, \"k8s.node.name\": nodeName } : {}\n ),\n // ...\n },\n});\n```\n\nwhich clears `empty_hostname` and attributes each run to its actual node.\n\n## Scope / risk\n\n- Additive: forwards one already-present, non-secret identity variable.\n- No behavior change when unset (existing `undefined` filter).\n- Aligns the k8s workload manager with the Docker one, which already forwards it.\n\n## Notes\n\nI have the exact change (plus a changeset) staged on a branch and am happy to open the PR once vouched — filing this as an issue first per the contributor policy. Reference branch/diff: PR #4213 (auto-closed by the unvouched-author check).\n","author":{"url":"https://github.com/brentshulman-silkline","@type":"Person","name":"brentshulman-silkline"},"datePublished":"2026-07-09T17:40:38Z","url":"https://github.com/triggerdotdev/trigger.dev/issues/4214"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:0fff1e61-4505-2dc1-a748-64924f0d04fc |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B516:137FBB:4A56F5E:639DAF9:6AFAE23F |
| html-safe-nonce | e7e8262a9a32e4fa521c51e7bae68f5051c0f636c1fe9fac0c0c1dac44d57ef7 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNTE2OjEzN0ZCQjo0QTU2RjVFOjYzOURBRjk6NkFGQUUyM0YiLCJ2aXNpdG9yX2lkIjoiMzc2OTAwNTIyMDQ3NDU3NzQ3MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | ddd304422ee64c9a9158294f4bcf6f8c0fff50652dfe781817d1936f6ef398ad |
| hovercard-subject-tag | issue:4848544211 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/triggerdotdev/trigger.dev/4214/issue_layout |
| twitter:image | https://opengraph.githubassets.com/102aa59efefdef9ecf1568cfe46889521bd9d74b89a22457c0a36cb88d2ab2f7/triggerdotdev/trigger.dev/issues/4214 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/102aa59efefdef9ecf1568cfe46889521bd9d74b89a22457c0a36cb88d2ab2f7/triggerdotdev/trigger.dev/issues/4214 |
| og:image:alt | Summary On self-hosted Kubernetes, task-run spans and logs exported over OTLP to an off-node collector (e.g. a centralized OpenTelemetry/Datadog Agent) arrive with no host. Datadog tags every such ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | brentshulman-silkline |
| hostname | github.com |
| expected-hostname | github.com |
| None | 9d9bd51a568909a83d1abb9d6f74d68f0c87ca85fe3a56d3f5d70d8b89add9f2 |
| turbo-cache-control | no-preview |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a7ae932999cc6d511008dbf980ff9efad2a31f7 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width