Title: TaskHubGrpcClient.schedule_new_orchestration records mismatched instanceId in OpenTelemetry producer span when instance_id is auto-generated · Issue #343 · microsoft/durabletask-python · GitHub
Open Graph Title: TaskHubGrpcClient.schedule_new_orchestration records mismatched instanceId in OpenTelemetry producer span when instance_id is auto-generated · Issue #343 · microsoft/durabletask-python
X Title: TaskHubGrpcClient.schedule_new_orchestration records mismatched instanceId in OpenTelemetry producer span when instance_id is auto-generated · Issue #343 · microsoft/durabletask-python
Description: Problem Description In TaskHubGrpcClient.schedule_new_orchestration (lines 639-650) and AsyncTaskHubGrpcClient.schedule_new_orchestration (lines 1187-1198), an instance ID is pre-resolved for the OpenTelemetry producer span: https://gith...
Open Graph Description: Problem Description In TaskHubGrpcClient.schedule_new_orchestration (lines 639-650) and AsyncTaskHubGrpcClient.schedule_new_orchestration (lines 1187-1198), an instance ID is pre-resolved for the O...
X Description: Problem Description In TaskHubGrpcClient.schedule_new_orchestration (lines 639-650) and AsyncTaskHubGrpcClient.schedule_new_orchestration (lines 1187-1198), an instance ID is pre-resolved for the O...
Opengraph URL: https://github.com/microsoft/durabletask-python/issues/343
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"TaskHubGrpcClient.schedule_new_orchestration records mismatched instanceId in OpenTelemetry producer span when instance_id is auto-generated","articleBody":"### Problem Description\n\nIn `TaskHubGrpcClient.schedule_new_orchestration` (lines 639-650) and `AsyncTaskHubGrpcClient.schedule_new_orchestration` (lines 1187-1198), an instance ID is pre-resolved for the OpenTelemetry producer span:\n\nhttps://github.com/microsoft/durabletask-python/blob/06bb27763eb893693f1bc300b86a86e01a5fa641/durabletask/client.py#L638-L650\n\n```python\n name = orchestrator if isinstance(orchestrator, str) else task.get_name(orchestrator)\n resolved_instance_id = instance_id if instance_id else uuid.uuid4().hex\n resolved_version = version if version else self.default_version\n\n with tracing.suppress_span_emission(not self._emit_trace_spans):\n with tracing.start_create_orchestration_span(\n name, resolved_instance_id, version=resolved_version,\n ):\n req = build_schedule_new_orchestration_req(\n orchestrator, input=input, instance_id=instance_id, start_at=start_at,\n reuse_id_policy=reuse_id_policy, tags=tags,\n version=version if version else self.default_version,\n data_converter=self._data_converter)\n```\n\nAnd similarly in `AsyncTaskHubGrpcClient`:\n\nhttps://github.com/microsoft/durabletask-python/blob/06bb27763eb893693f1bc300b86a86e01a5fa641/durabletask/client.py#L1186-L1198\n\n```python\n name = orchestrator if isinstance(orchestrator, str) else task.get_name(orchestrator)\n resolved_instance_id = instance_id if instance_id else uuid.uuid4().hex\n resolved_version = version if version else self.default_version\n\n with tracing.suppress_span_emission(not self._emit_trace_spans):\n with tracing.start_create_orchestration_span(\n name, resolved_instance_id, version=resolved_version,\n ):\n req = build_schedule_new_orchestration_req(\n orchestrator, input=input, instance_id=instance_id, start_at=start_at,\n reuse_id_policy=reuse_id_policy, tags=tags,\n version=version if version else self.default_version,\n data_converter=self._data_converter)\n```\n\nNotice that `resolved_instance_id` is passed to `start_create_orchestration_span()`, which records the attribute `durabletask.task.instance_id` on the OpenTelemetry `create_orchestration` producer span.\n\nHowever, `build_schedule_new_orchestration_req()` is passed `instance_id=instance_id` (which is `None` when relying on SDK auto-generation).\n\nInside `build_schedule_new_orchestration_req()`:\n\nhttps://github.com/microsoft/durabletask-python/blob/06bb27763eb893693f1bc300b86a86e01a5fa641/durabletask/internal/client_helpers.py#L86-L94\n\n```python\n return pb.CreateInstanceRequest(\n name=name,\n instanceId=instance_id if instance_id else uuid.uuid4().hex,\n ...\n )\n```\n\nBecause `instance_id` is `None`, a **second, completely different random UUID** is generated for `req.instanceId`.\n\nAs a result:\n1. `req.instanceId` is set to UUID #2.\n2. The client logs `Starting new '...' instance with ID = '{req.instanceId}'` with UUID #2.\n3. The sidecar creates the orchestration with UUID #2, which is returned by `schedule_new_orchestration()`.\n4. The worker processes the orchestration with UUID #2.\n5. **Meanwhile, the emitted OpenTelemetry `create_orchestration` span has `durabletask.task.instance_id` set to UUID #1 (`resolved_instance_id`), which does not exist anywhere in the backend.**\n\n---\n\n### Minimal Reproduction\n\n```python\nfrom unittest.mock import MagicMock\nfrom durabletask import client\nfrom durabletask.internal import tracing\n\ncaptured_attrs = {}\n\norig_start = tracing.start_create_orchestration_span\ndef mock_start(name, instance_id, version=None):\n captured_attrs[\"span_instance_id\"] = instance_id\n return orig_start(name, instance_id, version)\n\ntracing.start_create_orchestration_span = mock_start\n\nmock_stub = MagicMock()\nmock_stub.StartInstance.side_effect = lambda req: req\n\nc = client.TaskHubGrpcClient(channel=MagicMock())\nc._stub = mock_stub\n\n# Schedule new orchestration without explicit instance_id:\nactual_instance_id = c.schedule_new_orchestration(\"TestOrchestrator\")\n\nprint(\"Span instance_id: \", captured_attrs[\"span_instance_id\"])\nprint(\"Actual instance_id: \", actual_instance_id)\nprint(\"IDs match?: \", captured_attrs[\"span_instance_id\"] == actual_instance_id)\n```\n\nOutput:\n```\n2026-11-05 10:47:38.249 durabletask-client INFO: Starting new 'TestOrchestrator' instance with ID = 'f9bf3e536cdb44d0a074f3846cf7f1ef'.\nSpan instance_id: cda7391a06a24c84809b281ed7614531\nActual instance_id: f9bf3e536cdb44d0a074f3846cf7f1ef\nIDs match?: False\n```\n\n---\n\n### Expected Behavior\n\nWhen `instance_id` is not supplied by the caller, the OpenTelemetry producer span and the scheduled `CreateInstanceRequest` must share the exact same generated instance ID.\n\n---\n\n### Impact\n\nDistributed tracing correlation is broken for all orchestrations scheduled without explicit instance IDs (the primary default usage pattern). Queries in Application Insights, Datadog, Jaeger, Grafana Tempo, or OpenTelemetry Collector searching for the instance ID returned by `schedule_new_orchestration()` cannot correlate with the client's `create_orchestration` producer span because it is labeled with a phantom UUID.\n\n---\n\n### Suggested Direction\n\nPass `instance_id=resolved_instance_id` and `version=resolved_version` to `build_schedule_new_orchestration_req()` in both `TaskHubGrpcClient` and `AsyncTaskHubGrpcClient`:\n\n```python\n req = build_schedule_new_orchestration_req(\n orchestrator, input=input, instance_id=resolved_instance_id, start_at=start_at,\n reuse_id_policy=reuse_id_policy, tags=tags,\n version=resolved_version,\n data_converter=self._data_converter)\n```\n\n---\n\ncc @andystaples @berndverst\n","author":{"url":"https://github.com/hardikkaurani","@type":"Person","name":"hardikkaurani"},"datePublished":"2026-11-05T05:19:48Z","url":"https://github.com/microsoft/durabletask-python/issues/343"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:65ca0ec5-63df-3a2b-2e2e-6062ac67e878 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D8E4:2BFDE9:3147A7:40C3D7:6AFE3CDE |
| html-safe-nonce | 20282aff0a77a892c14ae1a0c8fca52b717b49fad2c6dcfbf3229d08731b54d8 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEOEU0OjJCRkRFOTozMTQ3QTc6NDBDM0Q3OjZBRkUzQ0RFIiwidmlzaXRvcl9pZCI6IjQ2MzI5MTIxNzgwOTU5OTYxMjYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | ea3354c28b2ae802521a4438a31cd699e7ccafbba3305b82c2a33ca1041fa963 |
| hovercard-subject-tag | issue:6217318282 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/343/issue_layout |
| twitter:image | https://opengraph.githubassets.com/29dff7b59e641f96606c4f7d3bab1aeba84425921d75cad7b75d5f129c93456e/microsoft/durabletask-python/issues/343 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/29dff7b59e641f96606c4f7d3bab1aeba84425921d75cad7b75d5f129c93456e/microsoft/durabletask-python/issues/343 |
| og:image:alt | Problem Description In TaskHubGrpcClient.schedule_new_orchestration (lines 639-650) and AsyncTaskHubGrpcClient.schedule_new_orchestration (lines 1187-1198), an instance ID is pre-resolved for the O... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | hardikkaurani |
| hostname | github.com |
| expected-hostname | github.com |
| None | f1a5901e2377b0af2a0b303f9528678f1475d04b9219dca1e993344196eaa07c |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | c4447fdce9ed0f33af959d2b6680d0bc18212a6a |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width