Title: Complete permanent payload failures with entity-lock cleanup for .NET parity · Issue #297 · microsoft/durabletask-python · GitHub
Open Graph Title: Complete permanent payload failures with entity-lock cleanup for .NET parity · Issue #297 · microsoft/durabletask-python
X Title: Complete permanent payload failures with entity-lock cleanup for .NET parity · Issue #297 · microsoft/durabletask-python
Description: Please add permanent orchestration payload-failure handling that releases entity locks, for parity with the fixes proposed in durabletask-dotnet PR 816. Source inspection of current main (cb75760f250099d5b3b49e18ed03e10f8fe46d0d) shows a...
Open Graph Description: Please add permanent orchestration payload-failure handling that releases entity locks, for parity with the fixes proposed in durabletask-dotnet PR 816. Source inspection of current main (cb75760f2...
X Description: Please add permanent orchestration payload-failure handling that releases entity locks, for parity with the fixes proposed in durabletask-dotnet PR 816. Source inspection of current main (cb75760f2...
Opengraph URL: https://github.com/microsoft/durabletask-python/issues/297
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Complete permanent payload failures with entity-lock cleanup for .NET parity","articleBody":"Please add permanent orchestration payload-failure handling that releases entity locks, for parity with the fixes proposed in [durabletask-dotnet PR 816](https://github.com/microsoft/durabletask-dotnet/pull/816). Source inspection of current `main` (`cb75760f250099d5b3b49e18ed03e10f8fe46d0d`) shows a related liveness/cleanup gap, not an identical action-clearing implementation.\r\n\r\n## Current behavior\r\n\r\n- [`TaskHubGrpcWorker._execute_orchestrator`](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/worker.py#L1278-L1293) externalizes the completed response before calling `CompleteOrchestratorTask`. If externalization raises, its catch block only logs the exception. That path sends neither a failed completion nor an explicit abandonment.\r\n- The configured maximum is a deterministic example: [`_try_externalize_field`](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L148-L177) raises `ValueError` when a raw payload exceeds `max_stored_payload_bytes`. Re-delivering the same work item cannot make that input fit the unchanged cap.\r\n- Entity cleanup exists in [`set_complete` / `set_failed`](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/worker.py#L1585-L1657): these release held locks and deliberately retain pending release actions. However, the later payload-error path does not use that cleanup. Even releases already included in a terminal response are not delivered if the response's result/custom status fails externalization.\r\n- The release generator, [`emit_lock_release_messages`](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/internal/orchestration_entity_context.py#L64-L76), contains the owner, critical-section ID, and target needed for each unlock.\r\n\r\nConsequently an orchestration can remain uncompleted and repeatedly encounter the same permanent payload error while previously acquired entity locks continue blocking other callers. Unlike the .NET bug, Python is not currently sending a terminal Failed response that has dropped its releases; it is not sending the completion at all.\r\n\r\n## Regression recipe\r\n\r\nUse the native worker with a fake payload store, a small threshold and maximum, and history showing an orchestration has acquired two entity locks. Cover both:\r\n\r\n1. A terminal orchestration response that already contains the two releases but has a result exceeding the store maximum.\r\n2. A pending entity call inside the critical section whose input exceeds that maximum; no releases have yet been emitted.\r\n\r\nRecord calls on the sidecar stub. The current `_execute_orchestrator` path logs the size exception without submitting either response. Expected parity behavior is a small non-retriable Failed completion containing the releases for both held locks, before the terminal action. A second caller should then be able to use the entities.\r\n\r\nThis report is based on the pinned source paths above; the recipe is requested regression coverage, not a claimed executed cloud test.\r\n\r\n## Acceptance Criteria\r\n\r\n- [ ] Deterministic permanent orchestration externalization failures, including the configured payload cap, do not leave the same work item in a log-only redelivery loop.\r\n- [ ] The worker submits a small non-retriable Failed completion, retaining already-emitted releases and generating any outstanding releases from acquired-lock state before sending.\r\n- [ ] Tests cover both terminal-result/custom-status failure and a pending entity-call failure with previously acquired locks; the release owner, target and critical-section IDs remain correct.\r\n- [ ] A backend-level regression demonstrates that another caller progresses after the failure releases its locks.\r\n- [ ] Transient storage failures and cancellation remain retryable/cancellable rather than being misclassified as permanent failures.\r\n- [ ] Preserve the existing token-idempotency behavior in both synchronous and asynchronous payload helpers.\r\n\r\n## Related\r\n\r\n- [.NET parity reference: PR 816](https://github.com/microsoft/durabletask-dotnet/pull/816), currently proposed rather than a released fix.\r\n- Python already skips recognized tokens before applying byte-size checks in the [sync helper](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L148-L177) and [async helper](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L277-L304). This issue does not request the .NET known-token fix where Python already has it.\r\n- [Issue 269](https://github.com/microsoft/durabletask-python/issues/269) concerns Azure Functions completion-size visibility/configuration. This report is specifically about native worker payload-store failure and entity-lock cleanup, not Functions configuration or Unicode serialization.\r\n","author":{"url":"https://github.com/YunchuWang","@type":"Person","name":"YunchuWang"},"datePublished":"2026-10-19T19:25:05Z","url":"https://github.com/microsoft/durabletask-python/issues/297"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:d32fe216-a882-36c2-38ed-d1ab0d49690e |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | ACC8:2AB11:CAAE14:11AB7D4:6AFE2CF1 |
| html-safe-nonce | 5df7deee06f25e6ffcb072bf515935395d675ef66e8fec4955fd7ab2bd6777c3 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQ0M4OjJBQjExOkNBQUUxNDoxMUFCN0Q0OjZBRkUyQ0YxIiwidmlzaXRvcl9pZCI6IjQwNTA3MDg4OTEzMDkwNTkzMTMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 0950ad1f23072300731ee655355d3d5037cb0e93a890a99caf053187e928d4e2 |
| hovercard-subject-tag | issue:5930129201 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/297/issue_layout |
| twitter:image | https://opengraph.githubassets.com/b0c6980621ff44444d310466f9ff8e6f21eb0f389d2716d636d5d65a5111e2a3/microsoft/durabletask-python/issues/297 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/b0c6980621ff44444d310466f9ff8e6f21eb0f389d2716d636d5d65a5111e2a3/microsoft/durabletask-python/issues/297 |
| og:image:alt | Please add permanent orchestration payload-failure handling that releases entity locks, for parity with the fixes proposed in durabletask-dotnet PR 816. Source inspection of current main (cb75760f2... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | YunchuWang |
| hostname | github.com |
| expected-hostname | github.com |
| None | e973f9ec3625f3faad8cac5c2d11a214b7ffe45f3cc563b828459b290ecaeed3 |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 1f92c367595b9190e0be293896078033886b1913 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width