René's URL Explorer Experiment


Title: Orphaned orchestration dispatches can indefinitely stall instances and exhaust a connected worker's capacity · Issue #304 · microsoft/durabletask-python · GitHub

Open Graph Title: Orphaned orchestration dispatches can indefinitely stall instances and exhaust a connected worker's capacity · Issue #304 · microsoft/durabletask-python

X Title: Orphaned orchestration dispatches can indefinitely stall instances and exhaust a connected worker's capacity · Issue #304 · microsoft/durabletask-python

Description: Summary If an orchestration dispatch exits without sending either a completion response or an abandonment request, the Python worker can remain alive and connected while the dispatch is no longer executing. The backend can continue retai...

Open Graph Description: Summary If an orchestration dispatch exits without sending either a completion response or an abandonment request, the Python worker can remain alive and connected while the dispatch is no longer e...

X Description: Summary If an orchestration dispatch exits without sending either a completion response or an abandonment request, the Python worker can remain alive and connected while the dispatch is no longer e...

Opengraph URL: https://github.com/microsoft/durabletask-python/issues/304

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Orphaned orchestration dispatches can indefinitely stall instances and exhaust a connected worker's capacity","articleBody":"## Summary\n\nIf an orchestration dispatch exits without sending either a completion response or an abandonment request, the Python worker can remain alive and connected while the dispatch is no longer executing. The backend can continue retaining ownership of that unfinished delivery, indefinitely stalling the affected instance and consuming server-side dispatch capacity.\n\nEnough orphaned deliveries can prevent the connection from receiving additional orchestration work even though the Python event loop and queue consumers are still running and local concurrency slots have been released.\n\nThis is a broader failure mode than the direct `asyncio.CancelledError` teardown case fixed by merged [#303](https://github.com/microsoft/durabletask-python/pull/303). Other execution exits, including grouped cancellation or another `BaseException` subclass, can still bypass both outcome paths. Related: [#301](https://github.com/microsoft/durabletask-python/issues/301) and [#300](https://github.com/microsoft/durabletask-python/pull/300).\n\n## Example trigger\n\nRegister and start an orchestrator whose dispatch-boundary finalizer raises an exception outside the existing `Exception` handlers:\n\n```python\nfrom durabletask import task\n\n\nclass TeardownExit(BaseException):\n    pass\n\n\ndef orchestrator(ctx: task.OrchestrationContext, _):\n    try:\n        yield ctx.call_activity(\"work\")\n    finally:\n        raise TeardownExit(\"dispatch teardown exited unexpectedly\")\n```\n\nThis illustrates the missing-outcome path without requiring an additional async framework. Equivalent failures can arise from a `BaseExceptionGroup` containing cancellation, or another runtime's cancellation exception. The problem is not limited to this particular exception class or to generator teardown: the relevant condition is an accepted dispatch ending without a scheduler-visible outcome.\n\n## Observed Python behavior\n\n- The exception escapes dispatch execution before the constructed response is returned or delivered.\n- Neither `CompleteOrchestratorTask` nor `AbandonTaskOrchestratorWorkItem` is called.\n- The per-work-item task still marks its local queue entry done and releases its semaphore in `finally`.\n- The queue consumer and work-item stream can remain healthy. Subsequent locally queued work can execute, so this is not inherently a Python event-loop or thread-pool deadlock.\n- This missing-outcome path does not itself initiate a reconnect.\n\nRelevant public SDK paths at the reviewed follow-up commit: [`_execute_orchestrator`](https://github.com/microsoft/durabletask-python/blob/92cb4123277d0d5a329e2436ec5d156f4359a57b/durabletask/worker.py#L1146-L1290), [generator teardown](https://github.com/microsoft/durabletask-python/blob/92cb4123277d0d5a329e2436ec5d156f4359a57b/durabletask/worker.py#L2351-L2368), and [work-item processing](https://github.com/microsoft/durabletask-python/blob/92cb4123277d0d5a329e2436ec5d156f4359a57b/durabletask/worker.py#L3698-L3764).\n\n## Consequences with Durable Task Scheduler\n\n- **Indefinitely stalled instance:** A healthy, open worker connection can retain and renew ownership of the unacknowledged delivery independently of Python execution liveness. Waiting for a nominal work-item timeout is therefore not sufficient to guarantee redispatch.\n- **No progress for that instance:** Additional events or task results can accumulate, but do not let another dispatch bypass the outstanding ownership for the same orchestration.\n- **No committed result from the lost response:** New actions, custom status, and terminal outcomes in that response are not committed. Previously committed history survives. An instance can remain Pending if its initial dispatch never committed, or retain its preceding state instead of becoming Failed.\n- **Connection-level orchestration starvation:** Orphaned deliveries retain backend dispatch capacity despite the released Python semaphore. One orphan need not exhaust the connection, but repeated occurrences can. The exact threshold is not simply the configured Python concurrency limit.\n- **Misleading worker health:** The worker can remain connected and continue handling health traffic while orchestration delivery is starved. Activity and entity delivery have separate capacity and need not stop merely because orchestration delivery is exhausted.\n- **Repeated failures can re-orphan work:** Recovering the connection alone does not resolve a deterministic execution failure encountered again on replay.\n\nThese backend consequences were established by source inspection; an end-to-end live-service reproduction has not been run. The Python-side continuation and missing abandonment were reproduced locally.\n\n## Recovery characteristics\n\nClosing the affected work-item stream or restarting its worker allows the backend to release ownership and redispatch pending work once the disconnect is observed. If normal disconnect cleanup cannot run, recovery depends on backend ownership expiry after renewal stops. No bounded recovery time is established here for an otherwise healthy connection that continues retaining the orphan.\n\nMerely waiting, adding another worker, sending an external event, or freeing local Python concurrency does not guarantee recovery of an already-owned orphaned instance. A graceful drain that leaves the old connection open is also not equivalent to closing it.\n\n## Expected behavior\n\nAn unexpected dispatch exit should not silently leave an instance owned indefinitely or consume enough delivery capacity to starve an otherwise healthy worker. There should be a defined, observable failure/recovery outcome for accepted work that is no longer executing.\n\nThis issue intentionally describes the failure conditions and consequences without prescribing an exception policy, ownership protocol, or implementation fix.","author":{"url":"https://github.com/andystaples","@type":"Person","name":"andystaples"},"datePublished":"2026-10-21T16:24:07Z","url":"https://github.com/microsoft/durabletask-python/issues/304"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:2b555e06-e181-3db4-c110-d38267eb10de
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idDC0C:1A0A35:7358D:9CC0F:6AFE2CD8
html-safe-noncef5b60a5f1c88cf09a9ee09b90cc779a9c671fa1cbde792b1a1cc691d0d2faa45
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEQzBDOjFBMEEzNTo3MzU4RDo5Q0MwRjo2QUZFMkNEOCIsInZpc2l0b3JfaWQiOiI3OTM1Njg0NTQ5MTcyNjY5NjU2IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacd72358e91099bdbc63806889cba90953d89f2c1795cd1a7f6b9497f0dddc2130
hovercard-subject-tagissue:5964371202
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/304/issue_layout
twitter:imagehttps://opengraph.githubassets.com/e579bf8a058c950e54649a87a6545d6ff0481b18905c5ef63f9b1908f63b085e/microsoft/durabletask-python/issues/304
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/e579bf8a058c950e54649a87a6545d6ff0481b18905c5ef63f9b1908f63b085e/microsoft/durabletask-python/issues/304
og:image:altSummary If an orchestration dispatch exits without sending either a completion response or an abandonment request, the Python worker can remain alive and connected while the dispatch is no longer e...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameandystaples
hostnamegithub.com
expected-hostnamegithub.com
Nonee973f9ec3625f3faad8cac5c2d11a214b7ffe45f3cc563b828459b290ecaeed3
turbo-cache-controlno-preview
go-importgithub.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git
octolytics-dimension-user_id6154722
octolytics-dimension-user_loginmicrosoft
octolytics-dimension-repository_id627664515
octolytics-dimension-repository_nwomicrosoft/durabletask-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id627664515
octolytics-dimension-repository_network_root_nwomicrosoft/durabletask-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release1f92c367595b9190e0be293896078033886b1913
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/microsoft/durabletask-python/issues/304#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fissues%2F304
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fissues%2F304
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=microsoft%2Fdurabletask-python
Reloadhttps://github.com/microsoft/durabletask-python/issues/304
Reloadhttps://github.com/microsoft/durabletask-python/issues/304
Reloadhttps://github.com/microsoft/durabletask-python/issues/304
microsofthttps://github.com/microsoft
durabletask-pythonhttps://github.com/microsoft/durabletask-python
Codehttps://github.com/microsoft/durabletask-python
Issues15 (15)https://github.com/microsoft/durabletask-python/issues
Pull requests12 (12)https://github.com/microsoft/durabletask-python/pulls
Actionshttps://github.com/microsoft/durabletask-python/actions
Projectshttps://github.com/microsoft/durabletask-python/projects
Security and qualityhttps://github.com/microsoft/durabletask-python/security
Insightshttps://github.com/microsoft/durabletask-python/insights
https://github.com/microsoft/durabletask-python
New issuehttps://github.com/login?return_to=https://github.com/microsoft/durabletask-python/issues/304
New issuehttps://github.com/login?return_to=https://github.com/microsoft/durabletask-python/issues/304
#307https://github.com/microsoft/durabletask-python/pull/307
Orphaned orchestration dispatches can indefinitely stall instances and exhaust a connected worker's capacityhttps://github.com/microsoft/durabletask-python/issues/304#top
#307https://github.com/microsoft/durabletask-python/pull/307
bugSomething isn't workinghttps://github.com/microsoft/durabletask-python/issues?q=state%3Aopen%20label%3A%22bug%22
https://github.com/andystaples
andystapleshttps://github.com/andystaples
on Oct 21, 2026https://github.com/microsoft/durabletask-python/issues/304#issue-5964371202
#303https://github.com/microsoft/durabletask-python/pull/303
#301https://github.com/microsoft/durabletask-python/issues/301
#300https://github.com/microsoft/durabletask-python/pull/300
_execute_orchestratorhttps://github.com/microsoft/durabletask-python/blob/92cb4123277d0d5a329e2436ec5d156f4359a57b/durabletask/worker.py#L1146-L1290
generator teardownhttps://github.com/microsoft/durabletask-python/blob/92cb4123277d0d5a329e2436ec5d156f4359a57b/durabletask/worker.py#L2351-L2368
work-item processinghttps://github.com/microsoft/durabletask-python/blob/92cb4123277d0d5a329e2436ec5d156f4359a57b/durabletask/worker.py#L3698-L3764
andystapleshttps://github.com/andystaples
bugSomething isn't workinghttps://github.com/microsoft/durabletask-python/issues?q=state%3Aopen%20label%3A%22bug%22
on Oct 21, 2026https://github.com/microsoft/durabletask-python/issues/304#event-33865869987
andystapleshttps://github.com/andystaples
completedhttps://github.com/microsoft/durabletask-python/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#307https://github.com/microsoft/durabletask-python/pull/307
on Oct 22, 2026https://github.com/microsoft/durabletask-python/issues/304#event-33970418552
Sign up for freehttps://github.com/signup?return_to=https://github.com/microsoft/durabletask-python/issues/304
Sign in to commenthttps://github.com/login?return_to=https://github.com/microsoft/durabletask-python/issues/304
bugSomething isn't workinghttps://github.com/microsoft/durabletask-python/issues?q=state%3Aopen%20label%3A%22bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.