René's URL Explorer Experiment


Title: Preflight all externalized payload sizes before starting uploads · Issue #298 · microsoft/durabletask-python · GitHub

Open Graph Title: Preflight all externalized payload sizes before starting uploads · Issue #298 · microsoft/durabletask-python

X Title: Preflight all externalized payload sizes before starting uploads · Issue #298 · microsoft/durabletask-python

Description: The synchronous and asynchronous payload helpers validate and upload one protobuf field at a time. A valid earlier field can therefore be uploaded and replaced by a token before a later field is rejected for a size that was already knowa...

Open Graph Description: The synchronous and asynchronous payload helpers validate and upload one protobuf field at a time. A valid earlier field can therefore be uploaded and replaced by a token before a later field is re...

X Description: The synchronous and asynchronous payload helpers validate and upload one protobuf field at a time. A valid earlier field can therefore be uploaded and replaced by a token before a later field is re...

Opengraph URL: https://github.com/microsoft/durabletask-python/issues/298

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Preflight all externalized payload sizes before starting uploads","articleBody":"The synchronous and asynchronous payload helpers validate and upload one protobuf field at a time. A valid earlier field can therefore be uploaded and replaced by a token before a later field is rejected for a size that was already knowable before any storage I/O. If that response is never submitted, the earlier upload has no backend-recorded reference.\r\n\r\n## Current source evidence\r\n\r\nInspected current `main` at [`cb75760f250099d5b3b49e18ed03e10f8fe46d0d`](https://github.com/microsoft/durabletask-python/commit/cb75760f250099d5b3b49e18ed03e10f8fe46d0d).\r\n\r\n- The [synchronous recursive walker](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L93-L170) visits fields and immediately calls `_try_externalize_field`, which checks only the current payload's maximum before uploading and mutating that field.\r\n- The [asynchronous walker](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L226-L300) has the same sequencing with `await store.upload_async(...)`.\r\n- The [worker completion path](https://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/worker.py#L1278-L1293) calls externalization before submitting `CompleteOrchestratorTask`; there is no whole-message preflight or compensation for successful earlier uploads in that path.\r\n\r\nThis is not a claim that the business request partially succeeds. The problem is the avoidable storage side effect before a deterministically invalid message fails. A cleanup process based on backend-recorded references cannot discover a reference that was never submitted or recorded.\r\n\r\n## Requested regression\r\n\r\nAdd this case to the existing `tests/durabletask/test_large_payload.py`, using its real protobuf types, helper and `FakePayloadStore`:\r\n\r\n```python\r\nstore = FakePayloadStore(threshold_bytes=8, max_stored_payload_bytes=32)\r\nresponse = pb.OrchestratorResponse(\r\n    instanceId=\"size-preflight\",\r\n    actions=[\r\n        pb.OrchestratorAction(\r\n            id=1,\r\n            scheduleTask=pb.ScheduleTaskAction(input=sv(\"a\" * 20)),\r\n        ),\r\n        pb.OrchestratorAction(\r\n            id=2,\r\n            scheduleTask=pb.ScheduleTaskAction(input=sv(\"b\" * 33)),\r\n        ),\r\n    ],\r\n)\r\nbefore = response.SerializeToString()\r\n\r\nwith pytest.raises(ValueError, match=\"exceeds the maximum\"):\r\n    externalize_payloads(response, store, instance_id=\"size-preflight\")\r\n\r\nassert len(store._blobs) == 0\r\nassert response.SerializeToString() == before\r\n```\r\n\r\nThe current source uploads the first input before inspecting the second input's maximum. Repeat the regression with `externalize_payloads_async`, plus an entity result with a valid earlier field and oversized later field.\r\n\r\n> [!NOTE]\r\n> This report is based on the pinned Python source and caller path. The Python regression above has not been executed for this filing. The equivalent JavaScript cases were reproduced and fixed with zero-upload assertions.\r\n\r\n## Acceptance Criteria\r\n\r\n- [ ] Collect and snapshot all relevant non-token payload fields before performing storage I/O, and validate their sizes according to the existing Python eligibility rules.\r\n- [ ] A deterministic maximum-size violation causes zero uploads and leaves all original protobuf payload values unchanged.\r\n- [ ] Apply the guarantee to both synchronous and asynchronous helpers, including nested/repeated messages, carryover events and entity payloads.\r\n- [ ] Preserve known-token idempotency, empty/absent fields, the existing strict-greater threshold behavior, and async cancellation/error propagation.\r\n- [ ] Preserve existing exception behavior for the helper; do not silently convert invalid input into a successful result.\r\n- [ ] Add valid-first/oversized-later, UTF-8 boundary, already-tokenized and all-valid regression cases.\r\n- [ ] Document the intentional sequencing change: deterministic invalid size is discovered before hypothetical earlier upload failures because no upload is attempted.\r\n- [ ] Avoid introducing new abstract members or mandatory dependencies for custom payload stores.\r\n\r\n## Non-goals\r\n\r\nThis does not introduce a cross-blob transaction, recover existing orphan blobs, or guarantee zero orphans after actual I/O/cancellation/submission failures. Unconditionally deleting uploads after an uncertain backend response is unsafe because the backend may already have accepted the reference.\r\n\r\nCarryover fields are already reached by Python's recursive protobuf traversal; this issue does not request a missing-field fix.\r\n\r\n## Related\r\n\r\n- [JavaScript zero-upload preflight fix](https://github.com/microsoft/durabletask-js/commit/55dfcf37da0ed3a2791f778d75621cb7f91144a7).\r\n- [.NET tracking issue 820](https://github.com/microsoft/durabletask-dotnet/issues/820).\r\n- [Python issue 297](https://github.com/microsoft/durabletask-python/issues/297) tracks permanent worker failure completion and entity-lock cleanup. Those acceptance criteria do not cover validating the entire message before any upload; this issue tracks the separate storage-side-effect guarantee.\r\n","author":{"url":"https://github.com/YunchuWang","@type":"Person","name":"YunchuWang"},"datePublished":"2026-10-19T21:01:17Z","url":"https://github.com/microsoft/durabletask-python/issues/298"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:caa5a95d-80c0-e9b9-7cb2-c4ba5d37ed71
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id9910:ADA14:C9817F:11995AA:6AFE2CE9
html-safe-noncee66cabf292e65bf5b994ffbf3fae9da7ecded5ca8fecc63e8cae9827f9518a83
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5OTEwOkFEQTE0OkM5ODE3RjoxMTk5NUFBOjZBRkUyQ0U5IiwidmlzaXRvcl9pZCI6IjU1NzQ5OTE1MzQ4MjM5NzYxNjkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacf302067873d9a67a62ac94999140b3067ea4c30a48f81320046756998e2dad1b
hovercard-subject-tagissue:5931503300
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/microsoft/durabletask-python/298/issue_layout
twitter:imagehttps://opengraph.githubassets.com/7664a700d81501da51483cfe4f6671cf69b0d36127a0706991605e872dabac5f/microsoft/durabletask-python/issues/298
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/7664a700d81501da51483cfe4f6671cf69b0d36127a0706991605e872dabac5f/microsoft/durabletask-python/issues/298
og:image:altThe synchronous and asynchronous payload helpers validate and upload one protobuf field at a time. A valid earlier field can therefore be uploaded and replaced by a token before a later field is re...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameYunchuWang
hostnamegithub.com
expected-hostnamegithub.com
Nonee973f9ec3625f3faad8cac5c2d11a214b7ffe45f3cc563b828459b290ecaeed3
turbo-cache-controlno-preview
go-importgithub.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git
octolytics-dimension-user_id6154722
octolytics-dimension-user_loginmicrosoft
octolytics-dimension-repository_id627664515
octolytics-dimension-repository_nwomicrosoft/durabletask-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id627664515
octolytics-dimension-repository_network_root_nwomicrosoft/durabletask-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release1f92c367595b9190e0be293896078033886b1913
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/microsoft/durabletask-python/issues/298#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fissues%2F298
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fissues%2F298
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=microsoft%2Fdurabletask-python
Reloadhttps://github.com/microsoft/durabletask-python/issues/298
Reloadhttps://github.com/microsoft/durabletask-python/issues/298
Reloadhttps://github.com/microsoft/durabletask-python/issues/298
microsofthttps://github.com/microsoft
durabletask-pythonhttps://github.com/microsoft/durabletask-python
Codehttps://github.com/microsoft/durabletask-python
Issues15 (15)https://github.com/microsoft/durabletask-python/issues
Pull requests12 (12)https://github.com/microsoft/durabletask-python/pulls
Actionshttps://github.com/microsoft/durabletask-python/actions
Projectshttps://github.com/microsoft/durabletask-python/projects
Security and qualityhttps://github.com/microsoft/durabletask-python/security
Insightshttps://github.com/microsoft/durabletask-python/insights
https://github.com/microsoft/durabletask-python
New issuehttps://github.com/login?return_to=https://github.com/microsoft/durabletask-python/issues/298
New issuehttps://github.com/login?return_to=https://github.com/microsoft/durabletask-python/issues/298
#305https://github.com/microsoft/durabletask-python/pull/305
Preflight all externalized payload sizes before starting uploadshttps://github.com/microsoft/durabletask-python/issues/298#top
#305https://github.com/microsoft/durabletask-python/pull/305
bugSomething isn't workinghttps://github.com/microsoft/durabletask-python/issues?q=state%3Aopen%20label%3A%22bug%22
https://github.com/YunchuWang
wangbill (YunchuWang)https://github.com/YunchuWang
on Oct 19, 2026https://github.com/microsoft/durabletask-python/issues/298#issue-5931503300
cb75760f250099d5b3b49e18ed03e10f8fe46d0dhttps://github.com/microsoft/durabletask-python/commit/cb75760f250099d5b3b49e18ed03e10f8fe46d0d
synchronous recursive walkerhttps://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L93-L170
asynchronous walkerhttps://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/payload/helpers.py#L226-L300
worker completion pathhttps://github.com/microsoft/durabletask-python/blob/cb75760f250099d5b3b49e18ed03e10f8fe46d0d/durabletask/worker.py#L1278-L1293
JavaScript zero-upload preflight fixhttps://github.com/microsoft/durabletask-js/commit/55dfcf37da0ed3a2791f778d75621cb7f91144a7
.NET tracking issue 820https://github.com/microsoft/durabletask-dotnet/issues/820
Python issue 297https://github.com/microsoft/durabletask-python/issues/297
wangbill (YunchuWang)https://github.com/YunchuWang
bugSomething isn't workinghttps://github.com/microsoft/durabletask-python/issues?q=state%3Aopen%20label%3A%22bug%22
on Oct 19, 2026https://github.com/microsoft/durabletask-python/issues/298#event-33685346566
https://github.com/chenyujia1024
CYJ (chenyujia1024)https://github.com/chenyujia1024
on Oct 21, 2026https://github.com/microsoft/durabletask-python/issues/298#issuecomment-6315126409
https://github.com/andystaples
andystapleshttps://github.com/andystaples
on Oct 21, 2026https://github.com/microsoft/durabletask-python/issues/298#issuecomment-6322135375
CYJ (@chenyujia1024)https://github.com/chenyujia1024
#305https://github.com/microsoft/durabletask-python/pull/305
Bernd Verst (berndverst)https://github.com/berndverst
completedhttps://github.com/microsoft/durabletask-python/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#305https://github.com/microsoft/durabletask-python/pull/305
on Oct 21, 2026https://github.com/microsoft/durabletask-python/issues/298#event-33895181811
Bernd Verst (berndverst)https://github.com/berndverst
on Oct 21, 2026https://github.com/microsoft/durabletask-python/issues/298#event-33895183729
Preflight all externalized payload sizes before uploads (#305)https://github.com/microsoft/durabletask-python/commit/100263bebe0e436e9e9f3055db1994855e1d1632
#305https://github.com/microsoft/durabletask-python/pull/305
https://docs.github.com/articles/closing-issues-via-commit-messages
100263bhttps://github.com/microsoft/durabletask-python/commit/100263bebe0e436e9e9f3055db1994855e1d1632
https://github.com/chenyujia1024
CYJ (chenyujia1024)https://github.com/chenyujia1024
on Oct 22, 2026https://github.com/microsoft/durabletask-python/issues/298#issuecomment-6345273235
Sign up for freehttps://github.com/signup?return_to=https://github.com/microsoft/durabletask-python/issues/298
Sign in to commenthttps://github.com/login?return_to=https://github.com/microsoft/durabletask-python/issues/298
bugSomething isn't workinghttps://github.com/microsoft/durabletask-python/issues?q=state%3Aopen%20label%3A%22bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.