Title: Executor does not close suspended orchestrator generators when a dispatch ends · Issue #299 · microsoft/durabletask-python · GitHub
Open Graph Title: Executor does not close suspended orchestrator generators when a dispatch ends · Issue #299 · microsoft/durabletask-python
X Title: Executor does not close suspended orchestrator generators when a dispatch ends · Issue #299 · microsoft/durabletask-python
Description: _OrchestrationExecutor.execute() reconstructs a generator for each dispatch, but does not explicitly close it when execution suspends at a durable wait. That can leave an async bridge's task, event loop, and orchestration context alive a...
Open Graph Description: _OrchestrationExecutor.execute() reconstructs a generator for each dispatch, but does not explicitly close it when execution suspends at a durable wait. That can leave an async bridge's task, event...
X Description: _OrchestrationExecutor.execute() reconstructs a generator for each dispatch, but does not explicitly close it when execution suspends at a durable wait. That can leave an async bridge's task, e...
Opengraph URL: https://github.com/microsoft/durabletask-python/issues/299
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Executor does not close suspended orchestrator generators when a dispatch ends","articleBody":"`_OrchestrationExecutor.execute()` reconstructs a generator for each dispatch, but does not explicitly close it when execution suspends at a durable wait. That can leave an async bridge's task, event loop, and orchestration context alive after the dispatch has finished.\r\n\r\nThis is more than delayed collection. An open AnyIO cancel scope is reachable through AnyIO's global weak-key task-state registry, whose value refers back to the task key. Calling `gc.collect()` does not release the suspended bridge.\r\n\r\n## Reproduction\r\n\r\nThe script below uses only the native SDK executor, AnyIO, and the standard library. It creates no scheduler connection and performs no real waits. Reproduced against `cb75760` on CPython 3.12.3 with `durabletask 1.12.0` and `anyio 4.14.2`.\r\n\r\nThe bridge yields a durable timer while an async task owns an AnyIO scope:\r\n\r\n```python\r\ndef episode(context):\r\n loop = asyncio.new_event_loop()\r\n\r\n async def body():\r\n assert context.instance_id == \"native-sdk-repro\"\r\n try:\r\n async with anyio.create_task_group():\r\n await asyncio.Future()\r\n finally:\r\n await asyncio.sleep(0)\r\n cleaned.append(\"closed\")\r\n\r\n root = loop.create_task(body())\r\n loop.run_until_complete(asyncio.sleep(0))\r\n try:\r\n yield context.create_timer(timedelta(seconds=5))\r\n finally:\r\n root.cancel()\r\n loop.run_until_complete(asyncio.gather(root, return_exceptions=True))\r\n loop.run_until_complete(loop.shutdown_asyncgens())\r\n loop.close()\r\n```\r\n\r\nAfter calling the native executor and collecting garbage, weak references to the context, generator, loop, and task are all still live, and `cleaned` is empty. Calling `generator.close()` explicitly runs the cleanup; a second close is harmless.\r\n\r\n```console\r\n$ python repro.py\r\nVersions: durabletask 1.12.0 anyio 4.14.2\r\nAfter execute + GC: {'loop': True, 'root': True, 'context': True, 'generator': True}\r\nCleanup: []\r\nAfter explicit idempotent close: ['closed']\r\n```\r\n\r\n**Expected:** the generator's cleanup finishes before the dispatch returns.\r\n\r\n**Actual:** its async task and context can remain reachable indefinitely, leaking one episode for each replay.\r\n\r\n\u003cdetails>\r\n\u003csummary>Full standalone reproducer\u003c/summary>\r\n\r\n```python\r\nimport asyncio\r\nimport gc\r\nimport logging\r\nfrom datetime import datetime, timedelta\r\nfrom importlib.metadata import version\r\nfrom weakref import ref\r\n\r\nimport anyio\r\nfrom durabletask import worker\r\nfrom durabletask.internal import helpers, orchestrator_service_pb2 as pb\r\nfrom durabletask.serialization import DEFAULT_DATA_CONVERTER\r\n\r\n\r\ndef main():\r\n references = {}\r\n cleaned = []\r\n\r\n def episode(context):\r\n loop = asyncio.new_event_loop()\r\n\r\n async def body():\r\n assert context.instance_id == 'native-sdk-repro'\r\n try:\r\n async with anyio.create_task_group():\r\n await asyncio.Future()\r\n finally:\r\n await asyncio.sleep(0)\r\n cleaned.append('closed')\r\n\r\n root = loop.create_task(body())\r\n references['loop'] = ref(loop)\r\n references['root'] = ref(root)\r\n loop.run_until_complete(asyncio.sleep(0))\r\n try:\r\n yield context.create_timer(timedelta(seconds=5))\r\n finally:\r\n root.cancel()\r\n loop.run_until_complete(asyncio.gather(root, return_exceptions=True))\r\n loop.run_until_complete(loop.shutdown_asyncgens())\r\n loop.close()\r\n\r\n def orchestrator(context, payload):\r\n generator = episode(context)\r\n references['context'] = ref(context)\r\n references['generator'] = ref(generator)\r\n return generator\r\n\r\n registry_worker = worker.TaskHubGrpcWorker.__new__(worker.TaskHubGrpcWorker)\r\n registry_worker._registry = worker._Registry()\r\n registry_worker._is_running = False\r\n registry_worker.add_orchestrator(orchestrator)\r\n started = pb.HistoryEvent(orchestratorStarted=pb.OrchestratorStartedEvent())\r\n started.timestamp.FromDatetime(datetime(2026, 1, 1))\r\n events = [\r\n started,\r\n helpers.new_execution_started_event('orchestrator', 'native-sdk-repro', None),\r\n ]\r\n try:\r\n worker._OrchestrationExecutor(\r\n registry_worker._registry, logging.getLogger('sdk-repro'), data_converter=DEFAULT_DATA_CONVERTER\r\n ).execute('native-sdk-repro', [], events)\r\n gc.collect()\r\n print('Versions:', 'durabletask', version('durabletask'), 'anyio', version('anyio'))\r\n print('After execute + GC:', {key: value() is not None for key, value in references.items()})\r\n print('Cleanup:', cleaned)\r\n assert cleaned == []\r\n assert all(value() is not None for value in references.values())\r\n finally:\r\n if (generator := references['generator']()) is not None:\r\n generator.close()\r\n generator.close()\r\n print('After explicit idempotent close:', cleaned)\r\n assert cleaned == ['closed']\r\n\r\n\r\nif __name__ == '__main__':\r\n main()\r\n```\r\n\r\n\u003c/details>\r\n\r\n## Suggested ownership boundary\r\n\r\nClose the generator in an executor `finally`, after all old/new events have been processed and the response has been constructed. Closing must not happen between events or allow teardown to add durable actions or change the already-decided response.\r\n\r\nLog ordinary cleanup failures with the instance and orchestration name, without replacing a dispatch result or masking version rejection. Cover suspension, completion, failure, continue-as-new, and exceptional exits, plus the AnyIO case with weak references and explicit collection.\r\n\r\nI hit this while building the Pydantic AI integration (pydantic/pydantic-ai#4764), which needs this guarantee from the SDK. The async/await work in #293 / #294 should use the same ownership boundary. I have a fix with tests ready and will open a PR referencing this issue.","author":{"url":"https://github.com/tamirdresher","@type":"Person","name":"tamirdresher"},"datePublished":"2026-10-20T11:04:21Z","url":"https://github.com/microsoft/durabletask-python/issues/299"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:5070995a-550f-f762-6f8f-26eea27cd28c |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 85F8:3E0418:6E36C:92B27:6AFE2CAF |
| html-safe-nonce | 3cb1f477cad87d27826c04266bd0bbc88efea1e1bf39e3bcb8d6f83809512779 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4NUY4OjNFMDQxODo2RTM2Qzo5MkIyNzo2QUZFMkNBRiIsInZpc2l0b3JfaWQiOiIxOTUxNTMyNjczMjIyNDU0NDQ3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | d8e4d4646b8578376c95332517f27171da1b708510fd3bf08f99064809f1601f |
| hovercard-subject-tag | issue:5940528337 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/299/issue_layout |
| twitter:image | https://opengraph.githubassets.com/1ec700e0bd7918603843281ef45d01b19e0fcb4b4cc6da3b500c26b297d740f1/microsoft/durabletask-python/issues/299 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/1ec700e0bd7918603843281ef45d01b19e0fcb4b4cc6da3b500c26b297d740f1/microsoft/durabletask-python/issues/299 |
| og:image:alt | _OrchestrationExecutor.execute() reconstructs a generator for each dispatch, but does not explicitly close it when execution suspends at a durable wait. That can leave an async bridge's task, event... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | tamirdresher |
| hostname | github.com |
| expected-hostname | github.com |
| None | e973f9ec3625f3faad8cac5c2d11a214b7ffe45f3cc563b828459b290ecaeed3 |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 1f92c367595b9190e0be293896078033886b1913 |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width