Title: [Async stack 2/6] asyncio composition and durable timers by andystaples · Pull Request #358 · microsoft/durabletask-python · GitHub
Open Graph Title: [Async stack 2/6] asyncio composition and durable timers by andystaples · Pull Request #358 · microsoft/durabletask-python
X Title: [Async stack 2/6] asyncio composition and durable timers by andystaples · Pull Request #358 · microsoft/durabletask-python
Description: WarningDo not merge this PR, or any other PR in this stack, on its own. The stack was split out of the closed #293 to make review easier. Once every PR in the stack is approved, the whole stack will be squash-merged together. Stack layer 2/6, built on the coroutine dispatch layer (#357). Summary This layer adds general asyncio composition and durable timers to the replay-safe event loop that coroutine orchestrators run on. Task composition Adds _DurableAsyncTask. A task is retired once its result or exception has been observed, so replaying discarded gather batches and handled failures does not keep decoded payloads or tracebacks alive. Tasks hash by creation sequence. TaskGroup iterates a set of tasks to cancel them, so this keeps cancellation order independent of memory addresses; the reason is documented in code. Cancellation callbacks run in a deterministic order, keyed by when each task was created. Retired tasks never reuse that ordering. TaskGroup, gather, wait_for, and asyncio.timeout compose over durable work: Cancelling a local waiter does not cancel the underlying shared durable operation. The actions emitted when a TaskGroup is cancelled stay the same across replays. A waiter finalized after the dispatch has closed (for example, an asyncio.Task constructed directly rather than through asyncio.create_task) no longer raises during garbage collection. Directly constructed tasks are documented as unsupported. Durable timers call_later and call_at are backed by durable timers and return _DurableTimerHandle, which compares by identity. Deadlines are rounded up to the next microsecond, so a timer never fires before the loop's deadline. Timezone-aware orchestration clocks are normalized. Deadlines already in the past run immediately, and non-finite delays raise an error. Cancelling a handle cancels its durable timer. asyncio.sleep and timeouts use the existing chunking for long timers. Unsupported work Every asyncio loop API the durable loop does not implement (executors, cross-thread scheduling, sockets, pipes, subprocesses, readers/writers, signal handlers, TLS, sendfile, task factories, exception handlers, debug mode, async-generator shutdown, and loop lifecycle) fails through one table with an actionable error, instead of a bare NotImplementedError. get_exception_handler() returns None, because callback failures always fail the dispatch. A test enumerates asyncio.AbstractEventLoop so that any public loop API that is neither implemented nor in the table, including one added by a future Python version, fails the suite. asyncio.wait and asyncio.as_completed are not rejected at run time; the contract documents them as unsupported because their ordering is not replay-stable. Docs (docs/async-orchestrators.md) States that successive dispatches can run on different workers and that all workers processing an instance must use the pinned environment. Notes that delays below the resolution of loop.time() are effectively zero, and that tasks must be created with asyncio.create_task or a task group. Code docs Extends the list of private asyncio hooks in the asyncio_loop.py module docstring, and clarifies the Task.wait() docstring for asyncio.create_task and task groups. Scope Left for later layers Dispatch teardown, async-generator hooks, the cleanup quiescence limit, and the cleanup error policy belong to layer 3/6 (#360). This layer keeps the minimal close() from layer 1/6, extended only to release the new timer and cancellation state. Orchestrators still receive the eager OrchestrationContext. The cold AsyncOrchestrationContext arrives in layer 4/6 (#361), which updates these tests to use it. No changelog changes; release notes are added in layer 6/6 (#364). Tests Adds tests/durabletask/test_asyncio_orchestrators.py, covering: composition memory retirement shared-wait cancellation timers unsupported work, including every unimplemented AbstractEventLoop API fanout scaling
Open Graph Description: WarningDo not merge this PR, or any other PR in this stack, on its own. The stack was split out of the closed #293 to make review easier. Once every PR in the stack is approved, the whole stack wil...
X Description: WarningDo not merge this PR, or any other PR in this stack, on its own. The stack was split out of the closed #293 to make review easier. Once every PR in the stack is approved, the whole stack wil...
Opengraph URL: https://github.com/microsoft/durabletask-python/pull/358
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:d75fd640-2c73-424c-0001-2b821d58307f |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 99E4:28801C:83B22:AE762:6AFE2D15 |
| html-safe-nonce | 723c240201000162a6360673551ef485c5a8c48d13cf4d3435ea1101990e30da |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5OUU0OjI4ODAxQzo4M0IyMjpBRTc2Mjo2QUZFMkQxNSIsInZpc2l0b3JfaWQiOiI2MTA3MDEzMDcyMjY0NTA2NjQ1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | cd513ed89398ccbc9875ce2bfe36a7c1ffd059aa5f57f6f6e73bddfdf9b7d47d |
| hovercard-subject-tag | pull_request:5265835794 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/microsoft/durabletask-python/pull/358/files |
| twitter:image | https://avatars.githubusercontent.com/u/77818326?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/77818326?s=400&v=4 |
| og:image:alt | WarningDo not merge this PR, or any other PR in this stack, on its own. The stack was split out of the closed #293 to make review easier. Once every PR in the stack is approved, the whole stack wil... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | e973f9ec3625f3faad8cac5c2d11a214b7ffe45f3cc563b828459b290ecaeed3 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 1f92c367595b9190e0be293896078033886b1913 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width