René's URL Explorer Experiment


Title: fix(worker): recover unexpected orchestration dispatch exits by andystaples · Pull Request #307 · microsoft/durabletask-python · GitHub

Open Graph Title: fix(worker): recover unexpected orchestration dispatch exits by andystaples · Pull Request #307 · microsoft/durabletask-python

X Title: fix(worker): recover unexpected orchestration dispatch exits by andystaples · Pull Request #307 · microsoft/durabletask-python

Description: Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancellation BaseException exits from execution, output serialization, or generator teardown produce a failed completion with the original type, message, and stack trace, rather than an orphan or deterministic abandonment loop. Output serialization keeps terminal callbacks non-reentrant, but rolls back the completion guard if serialization or completion-action construction exits. A custom to_json() failure creates a real FAILED action rather than acknowledging an empty response. ValueError/TypeError retain JsonEncodeOutputException; serialization StopIteration is a failure, not generator completion. Preserve an existing failure or actively escaping primary error. Ordinary teardown exceptions and direct teardown asyncio.CancelledError stay warning-only, as in #300/#303. Recognized cancellation/process-control signals, including nested built-in exception groups on Python 3.11+, still propagate. Guard the synchronous accepted-orchestration boundary. An exit before an outcome RPC attempts same-token abandonment once. If completion or abandonment raises/cancels, log the uncertain server-side outcome and cancel only the originating work-item stream, without retrying or sending a conflicting outcome. Bind recovery to the original stream for execution and queue-cancellation handlers. Cancelling an asyncio await does not abandon an executor thread that is still running. Reuse #306's full pre-teardown lock snapshot and ExecutionResults.as_failed for newly supported unexpected terminal failures, preserving pending-held obligations without changing ordinary replay actions or IDs. The typed permanent-payload fallback remains bounded and non-retriable. Verify literal legacy pending-call timeout history (unlock b at ID 3, activity at ID 4), then a later serialization or teardown failure emits only the missing unlock a before FAILED. Previously committed unlock b is not duplicated. Normal timeout replay retains historical actions. Update directly related documentation and package changelogs. No versions, dependency minimums, public abstract members, activity behavior, or entity dispatch policy change. This is recovery, not an exactly-once or bounded redispatch-time guarantee. Resetting a stream can permit redelivery of other work owned by that stream. Recognized control-flow signals are asyncio.CancelledError, GeneratorExit, SystemExit, and KeyboardInterrupt; unknown non-cancellation BaseException subclasses follow the explicit failure policy. Evidence and validation Final head: 3eee6632adb865f7db9c003954bca7a7a574c08e, based on parent-verified #306 f9942e2eb0f26098e296757c717a5189bf63c02c. All eight own commits were retained patch-equivalently through the final test-only lower follow-up (git range-diff). Published with an explicit lease against the recorded server-rebased top SHA. Same PR, refs, partial stack, and merged bottom member. Baseline #303 custom teardown reproduced zero completion and zero abandonment RPCs; the fix sends one FAILED completion and no abandonment. The human P2 output-serialization reproducer originally sent one completion with zero actions for both custom BaseException and RuntimeError, on direct and generator returns; those cases now contain FAILED with the original reason. Real local gRPC tests observe both teardown and serialization failures through the client and then complete another orchestration on the same concurrency-one worker. Check Final result Current-head GitHub CI All 27 checks pass; no failures or pending checks Core and DTS-emulator matrices Passed on Python 3.10–3.15 Azure Functions units Passed on Python 3.13–3.15 Real Functions-host E2E Passed in strict-typing modes 0 and 1 CI Pyright, lint, CodeQL Passed Local core unit scope 1,267 passed; 2 intentional direct-teardown skips covered by #303 Integrated recovery/cap/legacy-replay/executor/cleanup/tracing/resiliency scope 429 passed; 2 intentional skips Final test-only lower sync targeted integration 99 passed; 2 intentional skips Azure Managed service-free compatibility 182 passed Azure Functions service-free compatibility 408 passed Changed source/new recovery-test strict Pyright 0 errors, 0 warnings All three packages' source/test flake8 Passed Documentation Markdown lint, autopep8 diff, git diff check Passed Independent reviews prompted repairs for ambient exception-state handling, recovery-signal precedence, serialization callback reentrancy, and serialization StopIteration classification. The final narrow serialization/terminal-snapshot integration re-review reported no significant issues. The human P2 thread is fixed and resolved, and no inline review threads remain open. Earlier code-quality threads were addressed with valid simplifications or regression-based rationale where narrowing BaseException handling would reintroduce #304. The earlier Copilot review reported no open findings while appropriately requesting coordinated human review. Shared public SDK research supports an original-token outer dispatch boundary in .NET, but neither inspected .NET cancellation handling nor JavaScript task logging establishes Python's all-exits recovery invariant. No backend exactly-once guarantees are assumed. Real cloud-service ownership/recovery timing has not been live-validated. Emulator, local in-memory gRPC, and real Functions-host validation do not establish such guarantees.

Open Graph Description: Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancell...

X Description: Fixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancell...

Opengraph URL: https://github.com/microsoft/durabletask-python/pull/307

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:c71b7367-efd7-399b-7471-07107aec8c8b
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCA46:1BC582:CEA07A:1203CA0:6AFE2EE1
html-safe-nonce7935c7aaaa20d1638d105b64603fded3f7ef27632a5a5a496a61a5d14653f5f9
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQTQ2OjFCQzU4MjpDRUEwN0E6MTIwM0NBMDo2QUZFMkVFMSIsInZpc2l0b3JfaWQiOiI4ODU5ODE3MzQyMTA0NjQ1MzQ1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac6c7293a4405b8b104efbb92389fb0f0af7f8ac25988bdb3b55c99a931abd6cc9
hovercard-subject-tagpull_request:4955430852
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/microsoft/durabletask-python/pull/307/files
twitter:imagehttps://avatars.githubusercontent.com/u/77818326?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/77818326?s=400&v=4
og:image:altFixes #304 Depends on #306 in native stack 308: #305 (merged) → #306 → #307. Landing remains held for coordinated stack review and required human approval. Changes Previously unhandled non-cancell...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noned70e31220a67f6376c108d2416fa8e7df854c1065b1672ab7a88ab6dc3a809e2
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git
octolytics-dimension-user_id6154722
octolytics-dimension-user_loginmicrosoft
octolytics-dimension-repository_id627664515
octolytics-dimension-repository_nwomicrosoft/durabletask-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id627664515
octolytics-dimension-repository_network_root_nwomicrosoft/durabletask-python
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releaseccb88d9c92abe69245b429398c8a57ae87070c7c
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/microsoft/durabletask-python/pull/307/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fpull%2F307%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fpull%2F307%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=microsoft%2Fdurabletask-python
Reloadhttps://github.com/microsoft/durabletask-python/pull/307/files
Reloadhttps://github.com/microsoft/durabletask-python/pull/307/files
Reloadhttps://github.com/microsoft/durabletask-python/pull/307/files
microsofthttps://github.com/microsoft
durabletask-pythonhttps://github.com/microsoft/durabletask-python
Codehttps://github.com/microsoft/durabletask-python
Issues15 (15)https://github.com/microsoft/durabletask-python/issues
Pull requests12 (12)https://github.com/microsoft/durabletask-python/pulls
Actionshttps://github.com/microsoft/durabletask-python/actions
Projectshttps://github.com/microsoft/durabletask-python/projects
Security and qualityhttps://github.com/microsoft/durabletask-python/security
Insightshttps://github.com/microsoft/durabletask-python/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fmicrosoft%2Fdurabletask-python%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fmicrosoft%2Fdurabletask-python%2Fissues%2Fnew%2Fchoose
andystapleshttps://github.com/andystaples
mainhttps://github.com/microsoft/durabletask-python/tree/main
andystaples-recover-orphaned-orchestration-dispatchehttps://github.com/microsoft/durabletask-python/tree/andystaples-recover-orphaned-orchestration-dispatche
Conversation 21 https://github.com/microsoft/durabletask-python/pull/307
Commits 8 https://github.com/microsoft/durabletask-python/pull/307/commits
Checks 27 https://github.com/microsoft/durabletask-python/pull/307/checks
Files changed https://github.com/microsoft/durabletask-python/pull/307/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/307/files
fix(worker): recover unexpected orchestration dispatch exits https://github.com/microsoft/durabletask-python/pull/307/files#top
Show all changes 8 commits https://github.com/microsoft/durabletask-python/pull/307/files
c64dcb3 fix(worker): recover unexpected orchestration dispatch exits andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/c64dcb3a4118a3753982183b361267332ed18780
bc6a5e1 fix(worker): preserve recovery signals and stacked lock-release paths andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/bc6a5e1a47251f57c343ff4ac688b5e2ad55f1bd
96c6f03 test: simplify dispatch recovery parameter IDs andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/96c6f03e011cc51d8a91ff5486911c0219b9e561
4e27b58 test: preserve post-call release order in teardown failures andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/4e27b58f4163ec039d85cb18508748109f0ef9f4
ca85055 test: verify scheduler-visible teardown failure over local gRPC andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/ca85055d31ddf98b13e933d7b0430df78f4429da
2597983 test: cover compatible release order for pending entity calls andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/259798384a6449d47d53ea4efe20e3b21918dc61
fbeb42b fix(worker): recover output serialization exits before completion andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/fbeb42b1424ed12b990ddf4da616ed3a78f98094
d769d68 fix(worker): retain pending releases for unexpected terminal failures andystaples Oct 21, 2026 https://github.com/microsoft/durabletask-python/pull/307/commits/d769d68852c145dfc12f56423ae86260e8d8a367
Clear filters https://github.com/microsoft/durabletask-python/pull/307/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/307/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/307/files
CHANGELOG.md https://github.com/microsoft/durabletask-python/pull/307/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
CHANGELOG.md https://github.com/microsoft/durabletask-python/pull/307/files#diff-5238e244f2c6b5bd4ede65c19a2cd928c40ff1a5c28bf1979651e8fe1db1a6d2
supported-patterns.md https://github.com/microsoft/durabletask-python/pull/307/files#diff-ad1c71a2a02957a84278c89004cad2ce24d6a0f74b97413b72774f4eb20f266f
CHANGELOG.md https://github.com/microsoft/durabletask-python/pull/307/files#diff-1d1ece375471f083a849ea1b601985661b8bd8e7017fdb7fd5d10e02c4ca69f6
helpers.py https://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
worker.py https://github.com/microsoft/durabletask-python/pull/307/files#diff-821cca3b6d2d4c27b83bc2ea2c1dfbce058c46419967498ae26dc12693418899
test_worker_compat.py https://github.com/microsoft/durabletask-python/pull/307/files#diff-c1e9352e0d163ff9852246865936509240b49781a1ba7b85895c0ba9d797c997
test_azuremanaged_grpc_resiliency.py https://github.com/microsoft/durabletask-python/pull/307/files#diff-a08a2b1aedaa0c523550f379a73ba1cf0f0699e4234153adc78859158eae7034
test_orchestration_dispatch_recovery.py https://github.com/microsoft/durabletask-python/pull/307/files#diff-e0fd944ed9cc7b973593f53e8a60faaa91ac619ea7cbc3bdd8a1fb93d727a268
test_worker_resiliency.py https://github.com/microsoft/durabletask-python/pull/307/files#diff-9d331ea03310f3a8bc4964e5dd5e7b319f12a8ec210de6f90081f2f64befc55e
CHANGELOG.mdhttps://github.com/microsoft/durabletask-python/pull/307/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
View file https://github.com/microsoft/durabletask-python/blob/d769d68852c145dfc12f56423ae86260e8d8a367/CHANGELOG.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/307/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/307/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
https://github.com/microsoft/durabletask-python/pull/307/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
azure-functions-durable/CHANGELOG.mdhttps://github.com/microsoft/durabletask-python/pull/307/files#diff-5238e244f2c6b5bd4ede65c19a2cd928c40ff1a5c28bf1979651e8fe1db1a6d2
View file https://github.com/microsoft/durabletask-python/blob/d769d68852c145dfc12f56423ae86260e8d8a367/azure-functions-durable/CHANGELOG.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/307/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/307/files#diff-5238e244f2c6b5bd4ede65c19a2cd928c40ff1a5c28bf1979651e8fe1db1a6d2
https://github.com/microsoft/durabletask-python/pull/307/files#diff-5238e244f2c6b5bd4ede65c19a2cd928c40ff1a5c28bf1979651e8fe1db1a6d2
docs/supported-patterns.mdhttps://github.com/microsoft/durabletask-python/pull/307/files#diff-ad1c71a2a02957a84278c89004cad2ce24d6a0f74b97413b72774f4eb20f266f
View file https://github.com/microsoft/durabletask-python/blob/d769d68852c145dfc12f56423ae86260e8d8a367/docs/supported-patterns.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/307/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/307/files#diff-ad1c71a2a02957a84278c89004cad2ce24d6a0f74b97413b72774f4eb20f266f
https://github.com/microsoft/durabletask-python/pull/307/files#diff-ad1c71a2a02957a84278c89004cad2ce24d6a0f74b97413b72774f4eb20f266f
durabletask-azuremanaged/CHANGELOG.mdhttps://github.com/microsoft/durabletask-python/pull/307/files#diff-1d1ece375471f083a849ea1b601985661b8bd8e7017fdb7fd5d10e02c4ca69f6
View file https://github.com/microsoft/durabletask-python/blob/d769d68852c145dfc12f56423ae86260e8d8a367/durabletask-azuremanaged/CHANGELOG.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/307/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/307/files#diff-1d1ece375471f083a849ea1b601985661b8bd8e7017fdb7fd5d10e02c4ca69f6
https://github.com/microsoft/durabletask-python/pull/307/files#diff-1d1ece375471f083a849ea1b601985661b8bd8e7017fdb7fd5d10e02c4ca69f6
durabletask/internal/helpers.pyhttps://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
View file https://github.com/microsoft/durabletask-python/blob/d769d68852c145dfc12f56423ae86260e8d8a367/durabletask/internal/helpers.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/307/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
https://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
https://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
https://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
https://github.com/microsoft/durabletask-python/pull/307/files#diff-961116059c737297cf33e1a3e3ced16a6be891ef386862ae3a45337a550b5f97
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/307/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/307/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.