Title: fix(worker): recover unexpected orchestration dispatch exits by andystaples · Pull Request #307 · microsoft/durabletask-python · GitHub
Open Graph Title: fix(worker): recover unexpected orchestration dispatch exits by andystaples · Pull Request #307 · microsoft/durabletask-python
X Title: fix(worker): recover unexpected orchestration dispatch exits by andystaples · Pull Request #307 · microsoft/durabletask-python
Description: Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancellation BaseException exits from execution, output serialization, or generator teardown produce a failed completion with the original type, message, and stack trace, rather than an orphan or deterministic abandonment loop. Output serialization keeps terminal callbacks non-reentrant, but rolls back the completion guard if serialization or completion-action construction exits. A custom to_json() failure creates a real FAILED action rather than acknowledging an empty response. ValueError/TypeError retain JsonEncodeOutputException; serialization StopIteration is a failure, not generator completion. Preserve an existing failure or actively escaping primary error. Ordinary teardown exceptions and direct teardown asyncio.CancelledError stay warning-only, as in #300/#303. Recognized cancellation/process-control signals, including nested built-in exception groups on Python 3.11+, still propagate. Guard the synchronous accepted-orchestration boundary. An exit before an outcome RPC attempts same-token abandonment once. If completion or abandonment raises/cancels, log the uncertain server-side outcome and cancel only the originating work-item stream, without retrying or sending a conflicting outcome. Bind recovery to the original stream for execution and queue-cancellation handlers. Cancelling an asyncio await does not abandon an executor thread that is still running. Reuse #306's full pre-teardown lock snapshot and ExecutionResults.as_failed for newly supported unexpected terminal failures, preserving pending-held obligations without changing ordinary replay actions or IDs. The typed permanent-payload fallback remains bounded and non-retriable. Verify literal legacy pending-call timeout history (unlock b at ID 3, activity at ID 4), then a later serialization or teardown failure emits only the missing unlock a before FAILED. Previously committed unlock b is not duplicated. Normal timeout replay retains historical actions. Update directly related documentation and package changelogs. No versions, dependency minimums, public abstract members, activity behavior, or entity dispatch policy change. This is recovery, not an exactly-once or bounded redispatch-time guarantee. Resetting a stream can permit redelivery of other work owned by that stream. Recognized control-flow signals are asyncio.CancelledError, GeneratorExit, SystemExit, and KeyboardInterrupt; unknown non-cancellation BaseException subclasses follow the explicit failure policy. Evidence and validation Final head: 3eee6632adb865f7db9c003954bca7a7a574c08e, based on parent-verified #306 f9942e2eb0f26098e296757c717a5189bf63c02c. All eight own commits were retained patch-equivalently through the final test-only lower follow-up (git range-diff). Published with an explicit lease against the recorded server-rebased top SHA. Same PR, refs, partial stack, and merged bottom member. Baseline #303 custom teardown reproduced zero completion and zero abandonment RPCs; the fix sends one FAILED completion and no abandonment. The human P2 output-serialization reproducer originally sent one completion with zero actions for both custom BaseException and RuntimeError, on direct and generator returns; those cases now contain FAILED with the original reason. Real local gRPC tests observe both teardown and serialization failures through the client and then complete another orchestration on the same concurrency-one worker. Check Final result Current-head GitHub CI All 27 checks pass; no failures or pending checks Core and DTS-emulator matrices Passed on Python 3.10–3.15 Azure Functions units Passed on Python 3.13–3.15 Real Functions-host E2E Passed in strict-typing modes 0 and 1 CI Pyright, lint, CodeQL Passed Local core unit scope 1,267 passed; 2 intentional direct-teardown skips covered by #303 Integrated recovery/cap/legacy-replay/executor/cleanup/tracing/resiliency scope 429 passed; 2 intentional skips Final test-only lower sync targeted integration 99 passed; 2 intentional skips Azure Managed service-free compatibility 182 passed Azure Functions service-free compatibility 408 passed Changed source/new recovery-test strict Pyright 0 errors, 0 warnings All three packages' source/test flake8 Passed Documentation Markdown lint, autopep8 diff, git diff check Passed Independent reviews prompted repairs for ambient exception-state handling, recovery-signal precedence, serialization callback reentrancy, and serialization StopIteration classification. The final narrow serialization/terminal-snapshot integration re-review reported no significant issues. The human P2 thread is fixed and resolved, and no inline review threads remain open. Earlier code-quality threads were addressed with valid simplifications or regression-based rationale where narrowing BaseException handling would reintroduce #304. The earlier Copilot review reported no open findings while appropriately requesting coordinated human review. Shared public SDK research supports an original-token outer dispatch boundary in .NET, but neither inspected .NET cancellation handling nor JavaScript task logging establishes Python's all-exits recovery invariant. No backend exactly-once guarantees are assumed. Real cloud-service ownership/recovery timing has not been live-validated. Emulator, local in-memory gRPC, and real Functions-host validation do not establish such guarantees.
Open Graph Description: Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancell...
X Description: Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancell...
Opengraph URL: https://github.com/microsoft/durabletask-python/pull/307
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:c71b7367-efd7-399b-7471-07107aec8c8b |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | CA46:1BC582:CEA07A:1203CA0:6AFE2EE1 |
| html-safe-nonce | 7935c7aaaa20d1638d105b64603fded3f7ef27632a5a5a496a61a5d14653f5f9 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQTQ2OjFCQzU4MjpDRUEwN0E6MTIwM0NBMDo2QUZFMkVFMSIsInZpc2l0b3JfaWQiOiI4ODU5ODE3MzQyMTA0NjQ1MzQ1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 6c7293a4405b8b104efbb92389fb0f0af7f8ac25988bdb3b55c99a931abd6cc9 |
| hovercard-subject-tag | pull_request:4955430852 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/microsoft/durabletask-python/pull/307/files |
| twitter:image | https://avatars.githubusercontent.com/u/77818326?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/77818326?s=400&v=4 |
| og:image:alt | Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancell... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | d70e31220a67f6376c108d2416fa8e7df854c1065b1672ab7a88ab6dc3a809e2 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | ccb88d9c92abe69245b429398c8a57ae87070c7c |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width