Title: Python Durable Functions on DTS: oversized orchestrator completion fails the orchestration invisibly, and large-payload externalization can't be enabled · Issue #269 · microsoft/durabletask-python · GitHub
Open Graph Title: Python Durable Functions on DTS: oversized orchestrator completion fails the orchestration invisibly, and large-payload externalization can't be enabled · Issue #269 · microsoft/durabletask-python
X Title: Python Durable Functions on DTS: oversized orchestrator completion fails the orchestration invisibly, and large-payload externalization can't be enabled · Issue #269 · microsoft/durabletask-python
Description: Environment Azure Functions, Python 3.14, Elastic Premium, storageProvider.type: "azureManaged" (Durable Task Scheduler) azure-functions-durable==2.0.0rc1 (on durabletask==1.10.1), azure-functions==2.3.0 Durable extension 3.14.0 via Micr...
Open Graph Description: Environment Azure Functions, Python 3.14, Elastic Premium, storageProvider.type: "azureManaged" (Durable Task Scheduler) azure-functions-durable==2.0.0rc1 (on durabletask==1.10.1), azure-functions=...
X Description: Environment Azure Functions, Python 3.14, Elastic Premium, storageProvider.type: "azureManaged" (Durable Task Scheduler) azure-functions-durable==2.0.0rc1 (on durabletask==1.10.1), azure-...
Opengraph URL: https://github.com/microsoft/durabletask-python/issues/269
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Python Durable Functions on DTS: oversized orchestrator completion fails the orchestration invisibly, and large-payload externalization can't be enabled","articleBody":"### Environment\n\n- Azure Functions, Python 3.14, Elastic Premium, `storageProvider.type: \"azureManaged\"` (Durable Task Scheduler)\n- `azure-functions-durable==2.0.0rc1` (on `durabletask==1.10.1`), `azure-functions==2.3.0`\n- Durable extension 3.14.0 via `Microsoft.Azure.Functions.ExtensionBundle.Preview [4.29.0, 5.0.0)`\n\nWe have an instance ID and task-hub details for a production occurrence and can share them privately.\n\n### What happened\n\nAn orchestrator fanned out 121 activities; each result was 2–48 KB. When the last one completed, the orchestrator scheduled one more activity whose input was the fanned-in result list. The scheduler rejected that episode and marked the orchestration Failed:\n\n```\nruntimeStatus: FAILED\nfailureDetails.errorType: InvalidOperationException\nfailureDetails.message: The complete orchestration work item request of size 11.05MB exceeds the 3.90MB limit.\n Enable large-payload externalization to Azure Blob Storage to support oversized completion requests.\nExecutionCompletedEvent: orchestration_status 3, isNonRetriable: true\n```\n\n(Retrieved with `DurableTaskSchedulerClient.get_orchestration_state(fetch_payloads=True)` and `get_orchestration_history()`.)\n\n### Problems\n\n1. **The failure never reaches the app.** No exception surfaces in orchestrator code, so our `try/except` compensation (refund, user notification) never ran. The host and Application Insights log nothing. The last host line for the instance is `Orchestrator orchestrate_process_document yielded with 1 task(s) and 0 event(s) outstanding`. The failure exists only in scheduler state, and our users see a review stuck in \"processing\".\n2. **The rejected episode's triggering completion isn't in history.** History has 121 `TaskScheduled` events but only 120 `TaskCompleted`. The missing completion is the activity whose result triggered the rejected episode, and the host logged that activity as `Completed`. We assume this follows from the work item being rejected as a unit, but it means history and host logs disagree.\n3. **The 3.90 MB limit isn't documented.** The published DTS limits describe 1 MB per activity input/output. The cap on the orchestrator completion request, and the fact that scheduled-activity inputs count toward it, aren't documented anywhere we found. Our code had assumed the orchestrator→activity direction was safe.\n4. **The suggested fix isn't available to us.** The error message recommends large-payload externalization. `durabletask` 1.10.1 has it (`durabletask[azure-blob-payloads]`), and its worker externalizes `OrchestratorResponse` payloads when given a `payload_store`. But `azure-functions-durable` 2.0.0rc1 constructs `DurableFunctionsWorker` and `DurableFunctionsClient` without one, and exposes no way to pass one. The [large payload docs](https://learn.microsoft.com/en-us/azure/durable-task/scheduler/durable-task-scheduler-large-payloads) confirm Durable Functions support is .NET-isolated only.\n5. **Serialization inflates payloads for non-Latin text.** The stored activity results were 2.55 MB of UTF-8 (mostly Cyrillic), but the request was 11.05 MB. `azure.functions._durable_functions.df_dumps` calls `json.dumps` with the default `ensure_ascii=True`, so each Cyrillic character becomes a 6-byte `\\uXXXX` escape instead of 2 bytes of UTF-8.\n\n### Asks\n\n1. Deliver an oversized orchestrator completion as a failure the orchestrator can catch, or at least log it on the Functions host with the instance ID.\n2. Document the completion-request limit, and that scheduled-activity inputs count toward it.\n3. Is large-payload support planned for `azure-functions-durable` (Python) on DTS? We'd be glad to contribute a PR. Our rough plan:\n - accept a `PayloadStore` in `DFApp` or via app settings, and pass it to `DurableFunctionsWorker` and `DurableFunctionsClient`. The orchestrator and entity paths then externalize through the existing `durabletask` hooks, which alone would have prevented the failure above.\n - in the `activity_trigger` wrapper, resolve token inputs before calling the user function, and externalize outputs over the threshold. The open question is how the token should survive `df_dumps` JSON-quoting on the output side, so that `deexternalize_payloads` recognizes it in `TaskCompleted.result`. We'd like your preference before writing that part.\n4. Would you accept `ensure_ascii=False` in `df_dumps`? It's valid JSON and cuts non-Latin payloads about 3×, but it changes the bytes on the wire, so we'd rather ask first.\n","author":{"url":"https://github.com/nachogarcia","@type":"Person","name":"nachogarcia"},"datePublished":"2026-09-23T16:03:31Z","url":"https://github.com/microsoft/durabletask-python/issues/269"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ce1e523b-c9e5-53f1-1cac-6738085e8d18 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 907E:4E7F1:3A4E41:4F0CAB:6AFE1E8C |
| html-safe-nonce | cd7076a2e5c5eb1bed7c76e09056f1c4bc8fdd19cfdedc8a700084fbb547db4c |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MDdFOjRFN0YxOjNBNEU0MTo0RjBDQUI6NkFGRTFFOEMiLCJ2aXNpdG9yX2lkIjoiODY2ODUyNDkyNzU3OTQ2MzMwOCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | d0c9cdd7ab810d25de82d0ad5e58cce99dae8a86903902569028c90e96393ee2 |
| hovercard-subject-tag | issue:5556057847 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/269/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f71af6960c17c9bc7006a580681f8328ba92ab01268e81fa811add2d56a72015/microsoft/durabletask-python/issues/269 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f71af6960c17c9bc7006a580681f8328ba92ab01268e81fa811add2d56a72015/microsoft/durabletask-python/issues/269 |
| og:image:alt | Environment Azure Functions, Python 3.14, Elastic Premium, storageProvider.type: "azureManaged" (Durable Task Scheduler) azure-functions-durable==2.0.0rc1 (on durabletask==1.10.1), azure-functions=... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | nachogarcia |
| hostname | github.com |
| expected-hostname | github.com |
| None | d0525ab1464526bf4532a5551aa348cf100dff75ef808a96fc67a08b1d946aa9 |
| turbo-cache-control | no-preview |
| go-import | github.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git |
| octolytics-dimension-user_id | 6154722 |
| octolytics-dimension-user_login | microsoft |
| octolytics-dimension-repository_id | 627664515 |
| octolytics-dimension-repository_nwo | microsoft/durabletask-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 627664515 |
| octolytics-dimension-repository_network_root_nwo | microsoft/durabletask-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | fef3993a71fa20f7be8e167132fbf9619bc66e12 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width