René's URL Explorer Experiment


Title: Performance [P2]: Avoid eager worker imports from Azure managed package by berndverst · Pull Request #209 · microsoft/durabletask-python · GitHub

Open Graph Title: Performance [P2]: Avoid eager worker imports from Azure managed package by berndverst · Pull Request #209 · microsoft/durabletask-python

X Title: Performance [P2]: Avoid eager worker imports from Azure managed package by berndverst · Pull Request #209 · microsoft/durabletask-python

Description: Summary The durabletask package initializer eagerly imported durabletask.worker (and durabletask.payload.store) purely to re-export a handful of public types. Because durabletask-azuremanaged/durabletask/ has no __init__.py of its own — the Azure managed distribution shares the same durabletask namespace package — importing anything from it (e.g. durabletask.azuremanaged.client) first executes the core initializer, and therefore drags in the entire worker dependency graph (gRPC, protobuf, entities, serialization, OpenTelemetry) even for client-only applications. This PR resolves those public re-exports lazily via a module-level __getattr__ (PEP 562), paired with a TYPE_CHECKING block so static type checkers and IDEs still resolve every symbol. Resolved values are cached into the module globals, so only the first attribute access goes through the hook. All 10 re-exports are made lazy rather than just the worker ones — durabletask.payload.store triggers durabletask/payload/__init__.py, which imports payload/helpers.py, which imports google.protobuf. Same amount of code, strictly more benefit. Measurements Re-measured after merging main (2269c44). The module counts below are unchanged from the earlier measurement, so the packages added by #155, #198, #206, #210 and #211 did not shift the baseline. python -X importtime, medians of 9 runs. Times are the cumulative column of the module's own importtime line — i.e. the cost attributable to the import statement, excluding interpreter startup. Bare interpreter startup on this machine is 93.3 ms / 78 modules, which is the floor for the module counts. Scenario Before After Delta import durabletask — attributable time 415.0 ms 5.7 ms ~73x faster import durabletask — modules resident 294 82 −212 modules import durabletask.azuremanaged.client — attributable time 716.8 ms 646.2 ms ~70 ms import durabletask.azuremanaged.client — modules resident 409 399 −10 modules ImportantThe headline "415 ms → 5.7 ms" applies to a bare import durabletask. The realistic client path, import durabletask.azuremanaged.client, improves far less — roughly 70 ms and 10 modules — because that module imports gRPC and protobuf directly for its own purposes, so this change cannot avoid them. The large win accrues to code that imports durabletask (or anything under the namespace) without needing the worker; the client path benefits only modestly. Modules no longer loaded by import durabletask.azuremanaged.client: durabletask.worker, durabletask.internal.exceptions, durabletask.internal.json_encode_output_exception, durabletask.internal.orchestration_entity_context, durabletask.internal.type_discovery, packaging, packaging.version, queue, _queue, concurrent.futures.thread. NoteThis machine runs many concurrent workloads, so cold-import timings vary by roughly ±10% run to run. The module counts are exact and deterministic, and are the more reliable signal. Interaction with #197 #197 makes the same kind of change for the sandboxes preview package. Measuring import durabletask.azuremanaged.preview.sandboxes across all four combinations shows the two changes are super-additive — neither alone gets close to the combined result. (These figures are totals including the interpreter floor, and were independently reproduced by the #197 author with identical module counts in all four cells.) Configuration Time Modules Neither change 1018.0 ms 548 This PR (#196) alone 1040.4 ms 548 (no change) #197 alone 467.5 ms 296 Both 94.0 ms 84 To be explicit about what this PR does not do: on its own it gives no improvement to the sandboxes import path, because sandboxes/__init__.py independently imports the worker graph, so that graph loads regardless of what the core initializer does. Symmetrically, #197 alone still pays for the eager core initializer. Only closing both doors avoids loading gRPC/protobuf at all. The two PRs are complementary and independently correct; neither blocks the other, and they touch disjoint files. Backwards compatibility No public API changes. Explicitly preserved and covered by tests: Every existing import path: from durabletask import ConcurrencyOptions, GrpcChannelOptions, GrpcRetryPolicyOptions, LargePayloadStorageOptions, PayloadStore, ActivityWorkItemFilter, EntityWorkItemFilter, OrchestrationWorkItemFilter, VersioningOptions, WorkItemFilters __all__ verbatim (same names, same order) and PACKAGE_NAME dir(durabletask) still returns the public names (module-level __dir__ added) from durabletask import * Submodule attribute access after a bare import durabletask — see below Submodule imports (from durabletask import task, client, worker, entities) — these keep working because __getattr__ raises AttributeError, letting importlib's _handle_fromlist fall back to importing the submodule The exact AttributeError message shape for genuinely unknown attributes Each lazily-resolved symbol is the same object as in its defining module (identity, not a copy) No import cycle: durabletask/worker.py does from durabletask import task, and both import orders (import durabletask first, import durabletask.worker first) are verified. Preserving submodule attribute bindings The eager from durabletask.worker import ... had a second, easily-missed effect: importing a submodule binds it as an attribute of its parent package. So on main, a bare import durabletask left seven submodules reachable as attributes — entities, grpc_options, internal, payload, serialization, task, worker — and code doing import durabletask followed by durabletask.task.X worked. Making the re-exports lazy removed that side effect, so those attributes started raising AttributeError. This was caught in review and is fixed here by a _LAZY_SUBMODULES frozenset that __getattr__ consults after the export lookup misses, resolving via import_module(f".{name}", __name__) and caching the result the same way. The names are also included in __dir__. This stays fully lazy — nothing is imported at module load, so the cold-start win is unaffected (re-measured post-merge: 5.7 ms, 82 modules, with durabletask.worker and grpc still absent from sys.modules). Two details worth noting: The seven names were derived empirically from a clean main checkout rather than hand-picked, so the restored surface matches exactly. durabletask.client is deliberately not included. It was never transitively imported on main, so it already failed this way; adding it would expand the public surface rather than preserve it. A test pins that it stays unbound. A full attribute-surface comparison against main confirms nothing is lost. Compatibility with the new azure-functions-durable package main added the in-repo azure-functions-durable package (#155), which imports heavily from this namespace. All 26 of its distinct durabletask imports use the explicit from X import Y form, which resolves correctly under PEP 562; every other textual durabletask. occurrence is a docstring cross-reference or a deprecation message, not runtime attribute access. Verified empirically rather than assumed — its suite returns the same 202 passed both with this change and with main's eager initializer. CI fix The first CI run failed on all five Python versions with ModuleNotFoundError: No module named 'durabletask.azuremanaged' in test_importing_azuremanaged_client_does_not_import_worker. The cause is a missing package, not lazy initialization: the run-tests job in .github/workflows/durabletask.yml installs requirements.txt, .[azure-blob-payloads] and aiohttp, and never installs durabletask-azuremanaged. The test's subprocess could not import the provider there under any implementation; it passed locally only because a development environment has both distributions installed. Fixed by guarding that one test with pytest.importorskip("durabletask.azuremanaged"), so it skips cleanly where the provider is absent and still runs — with its assertion unchanged — wherever it is present. Widening the workflow's install scope seemed out of scope for this PR. Verified by blocking the provider behind a sys.meta_path hook to simulate the CI environment: 42 passed / 1 skipped without the provider, 43 passed with it. Verification All commands run against a session-local venv with the packages installed editable from this worktree, after merging main (0f316cc). Check Result pytest tests/durabletask -q --ignore-glob="*_e2e.py" 727 passed, 7 skipped pytest tests/azure-functions-durable -m "not dts and not azurite and not functions_e2e" 202 passed — identical to the same run against main's eager initializer pytest tests/durabletask-azuremanaged/test_sandboxes_extension.py tests/durabletask-azuremanaged/test_durabletask_grpc_interceptor.py -q 45 passed flake8 . (as the lint job runs it) clean, exit 0 pyright strict on durabletask/__init__.py 0 diagnostics pymarkdown -c .pymarkdown.json scan CHANGELOG.md 0 findings on the added entry New tests in tests/durabletask/test_lazy_exports.py (43 tests) cover: durabletask.worker is not in sys.modules after a bare import durabletask (run in a subprocess for isolation), and likewise for grpc, google.protobuf, and opentelemetry The same holds after import durabletask.azuremanaged.client Each of the seven eagerly-bound submodules resolves by attribute access after a bare import durabletask, and does so lazily Every public name is importable, resolves to the same object as in its defining module, and its source module is pinned independently of the implementation __all__, dir(), star-import, PACKAGE_NAME, AttributeError shape, caching, and both import orders Sanity check that the tests are meaningful: the submodule regression tests were written first and confirmed to fail (8 failures) against the unfixed implementation. Likewise, against the original eager initializer the lazy-behavior tests fail while the backwards-compatibility tests pass on both implementations — which is the intended split, since the compat tests pin behavior that must not change. Note on the CHANGELOG entry CHANGELOG.md has 16 pre-existing MD013 line-length violations on main. The entry added here is wrapped to the 100-character limit using non-indented continuation lines, keeping that count unchanged rather than adding to it. Reflowing the rest of the file seemed out of scope. Fixes #196

Open Graph Description: Summary The durabletask package initializer eagerly imported durabletask.worker (and durabletask.payload.store) purely to re-export a handful of public types. Because durabletask-azuremanaged/durab...

X Description: Summary The durabletask package initializer eagerly imported durabletask.worker (and durabletask.payload.store) purely to re-export a handful of public types. Because durabletask-azuremanaged/durab...

Opengraph URL: https://github.com/microsoft/durabletask-python/pull/209

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:89527f45-7f32-89a1-65b6-dbf3f4bb1e3d
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id9958:B0D00:38F8D4:4CE39F:6AFE414E
html-safe-nonce8f9c409ef7ddf29af21997ffb4870c9a1916907c673e54eb7e2cde261a02cb19
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5OTU4OkIwRDAwOjM4RjhENDo0Q0UzOUY6NkFGRTQxNEUiLCJ2aXNpdG9yX2lkIjoiNzk5MTU2Njk1NzA3MTQ1MDQ0NiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac90623db34c0f0ebcf8bd3e79a29c99dff5fe92916a4ddff16f59b967b775fcc3
hovercard-subject-tagpull_request:4139850919
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/microsoft/durabletask-python/pull/209/files
twitter:imagehttps://avatars.githubusercontent.com/u/4535280?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/4535280?s=400&v=4
og:image:altSummary The durabletask package initializer eagerly imported durabletask.worker (and durabletask.payload.store) purely to re-export a handful of public types. Because durabletask-azuremanaged/durab...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Nonef1a5901e2377b0af2a0b303f9528678f1475d04b9219dca1e993344196eaa07c
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/microsoft/durabletask-python git https://github.com/microsoft/durabletask-python.git
octolytics-dimension-user_id6154722
octolytics-dimension-user_loginmicrosoft
octolytics-dimension-repository_id627664515
octolytics-dimension-repository_nwomicrosoft/durabletask-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id627664515
octolytics-dimension-repository_network_root_nwomicrosoft/durabletask-python
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasec4447fdce9ed0f33af959d2b6680d0bc18212a6a
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/microsoft/durabletask-python/pull/209/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fpull%2F209%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fdurabletask-python%2Fpull%2F209%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=microsoft%2Fdurabletask-python
Reloadhttps://github.com/microsoft/durabletask-python/pull/209/files
Reloadhttps://github.com/microsoft/durabletask-python/pull/209/files
Reloadhttps://github.com/microsoft/durabletask-python/pull/209/files
microsofthttps://github.com/microsoft
durabletask-pythonhttps://github.com/microsoft/durabletask-python
Codehttps://github.com/microsoft/durabletask-python
Issues15 (15)https://github.com/microsoft/durabletask-python/issues
Pull requests12 (12)https://github.com/microsoft/durabletask-python/pulls
Actionshttps://github.com/microsoft/durabletask-python/actions
Projectshttps://github.com/microsoft/durabletask-python/projects
Security and qualityhttps://github.com/microsoft/durabletask-python/security
Insightshttps://github.com/microsoft/durabletask-python/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fmicrosoft%2Fdurabletask-python%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fmicrosoft%2Fdurabletask-python%2Fissues%2Fnew%2Fchoose
Bernd Verst (berndverst)https://github.com/berndverst
mainhttps://github.com/microsoft/durabletask-python/tree/main
berndverst-lazy-worker-exports-issue-196https://github.com/microsoft/durabletask-python/tree/berndverst-lazy-worker-exports-issue-196
Conversation 18 https://github.com/microsoft/durabletask-python/pull/209
Commits 8 https://github.com/microsoft/durabletask-python/pull/209/commits
Checks 24 https://github.com/microsoft/durabletask-python/pull/209/checks
Files changed https://github.com/microsoft/durabletask-python/pull/209/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/209/files
Performance [P2]: Avoid eager worker imports from Azure managed package https://github.com/microsoft/durabletask-python/pull/209/files#top
Show all changes 8 commits https://github.com/microsoft/durabletask-python/pull/209/files
07a7d40 Avoid eager worker imports from the durabletask package initializer Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/07a7d404d836201a56efcaafc13c5557794e7d2a
fdd2b99 Keep eagerly bound submodules reachable as package attributes Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/fdd2b99c1a21222499e3c2ac8e265c11e3d9c306
89f56c1 Merge remote-tracking branch 'origin/main' into berndverst-lazy-worke… Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/89f56c1b306e53ac603512bb7f073b9c01084e7e
ea332e9 Skip the azuremanaged lazy-import test when the provider is absent Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/ea332e91447709ddadc5115756e2d8fc7f030a51
e1db0f4 Indent CHANGELOG continuation lines for repo consistency Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/e1db0f42497b9ee655f443eb36330df0ca41515a
08207b1 Merge remote-tracking branch 'origin/main' into berndverst-lazy-worke… Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/08207b1f63a93d8730c7fd76d1947de4ef927df2
1ea92d6 Restore unindented changelog continuation lines Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/1ea92d66a4184fa0efe1e4dccf31c4adc91abecc
8738e1d Merge remote-tracking branch 'origin/main' into berndverst-lazy-worke… Jul 27, 2026 https://github.com/microsoft/durabletask-python/pull/209/commits/8738e1d6e61ef75e8976c44f1d093ab2a66fe816
Clear filters https://github.com/microsoft/durabletask-python/pull/209/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/209/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/209/files
CHANGELOG.md https://github.com/microsoft/durabletask-python/pull/209/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
__init__.py https://github.com/microsoft/durabletask-python/pull/209/files#diff-8c47fef5e17c3e30d3d5326eb5c5bc04ee283701e3d2fe56d00537f3834dffff
test_lazy_exports.py https://github.com/microsoft/durabletask-python/pull/209/files#diff-2dc378dedd7e1ae7c7c35a92369c739c0edd2b36dffb170b9532b9827d83aa40
CHANGELOG.mdhttps://github.com/microsoft/durabletask-python/pull/209/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
View file https://github.com/microsoft/durabletask-python/blob/8738e1d6e61ef75e8976c44f1d093ab2a66fe816/CHANGELOG.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/209/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/209/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
https://github.com/microsoft/durabletask-python/pull/209/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed
durabletask/__init__.pyhttps://github.com/microsoft/durabletask-python/pull/209/files#diff-8c47fef5e17c3e30d3d5326eb5c5bc04ee283701e3d2fe56d00537f3834dffff
View file https://github.com/microsoft/durabletask-python/blob/8738e1d6e61ef75e8976c44f1d093ab2a66fe816/durabletask/__init__.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/209/{{ revealButtonHref }}
https://github.com/microsoft/durabletask-python/pull/209/files#diff-8c47fef5e17c3e30d3d5326eb5c5bc04ee283701e3d2fe56d00537f3834dffff
https://github.com/microsoft/durabletask-python/pull/209/files#diff-8c47fef5e17c3e30d3d5326eb5c5bc04ee283701e3d2fe56d00537f3834dffff
tests/durabletask/test_lazy_exports.pyhttps://github.com/microsoft/durabletask-python/pull/209/files#diff-2dc378dedd7e1ae7c7c35a92369c739c0edd2b36dffb170b9532b9827d83aa40
View file https://github.com/microsoft/durabletask-python/blob/8738e1d6e61ef75e8976c44f1d093ab2a66fe816/tests/durabletask/test_lazy_exports.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/durabletask-python/pull/209/{{ revealButtonHref }}
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/209/files
Please reload this pagehttps://github.com/microsoft/durabletask-python/pull/209/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.