René's URL Explorer Experiment


Title: fix(throttling-manager): Open the per-domain sub-queues on first use instead of on insert by Mantisus · Pull Request #2177 · apify/crawlee-python · GitHub

Open Graph Title: fix(throttling-manager): Open the per-domain sub-queues on first use instead of on insert by Mantisus · Pull Request #2177 · apify/crawlee-python

X Title: fix(throttling-manager): Open the per-domain sub-queues on first use instead of on insert by Mantisus · Pull Request #2177 · apify/crawlee-python

Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included. - fix(throttling-manager): Open the per-domain sub-queues on first use instead of on insert by Mantisus · Pull Request #2177 · apify/crawlee-python

Open Graph Description: Description A sub-manager for every configured domain is now opened on first use, instead of only when a request for that domain is added. The read paths used to iterate an empty dict on a fresh p...

X Description: Description A sub-manager for every configured domain is now opened on first use, instead of only when a request for that domain is added. The read paths used to iterate an empty dict on a fresh p...

Opengraph URL: https://github.com/apify/crawlee-python/pull/2177

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:2b606a3e-dc9c-95c4-006b-07d5cf7cd6d8
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idE170:21E3E7:C8E075:10A65B5:6B00F5AA
html-safe-nonce72b174f10ab3b701aafb976aade8f1497955196433b3318dab998d223fb67b25
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMTcwOjIxRTNFNzpDOEUwNzU6MTBBNjVCNTo2QjAwRjVBQSIsInZpc2l0b3JfaWQiOiIyMzYzNjI2NjY3Nzg1NjQ3NTMxIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac1cc84ab4f246fd7c6c39d42ecaecc65a08a903b796bfd5a98b9df3a083caba3d
hovercard-subject-tagpull_request:4317632219
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/apify/crawlee-python/2177/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/02a4cc66d46eedb08e841db3a4dc30a775f65147bf809859c5953612be10ea6e/apify/crawlee-python/pull/2177
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/02a4cc66d46eedb08e841db3a4dc30a775f65147bf809859c5953612be10ea6e/apify/crawlee-python/pull/2177
og:image:altDescription A sub-manager for every configured domain is now opened on first use, instead of only when a request for that domain is added. The read paths used to iterate an empty dict on a fresh p...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameMantisus
hostnamegithub.com
expected-hostnamegithub.com
Nonee208d86428ec00a57ade75058166297440e8128a01e5287c31a53dc042bbf187
turbo-cache-controlno-cache
go-importgithub.com/apify/crawlee-python git https://github.com/apify/crawlee-python.git
octolytics-dimension-user_id24586296
octolytics-dimension-user_loginapify
octolytics-dimension-repository_id741365632
octolytics-dimension-repository_nwoapify/crawlee-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id741365632
octolytics-dimension-repository_network_root_nwoapify/crawlee-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release7dafd017fb7809a839c443dba08622af38e63c09
ui-targetcanary-1
theme-color#1e2327
color-schemelight dark
react-app-namepull-requests

Links:

Skip to contenthttps://github.com/apify/crawlee-python/pull/2177#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2177
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2177
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=apify%2Fcrawlee-python
Reloadhttps://github.com/apify/crawlee-python/pull/2177
Reloadhttps://github.com/apify/crawlee-python/pull/2177
Reloadhttps://github.com/apify/crawlee-python/pull/2177
apifyhttps://github.com/apify
crawlee-pythonhttps://github.com/apify/crawlee-python
Codehttps://github.com/apify/crawlee-python
Issues72 (72)https://github.com/apify/crawlee-python/issues
Pull requests14 (14)https://github.com/apify/crawlee-python/pulls
Discussionshttps://github.com/apify/crawlee-python/discussions
Actionshttps://github.com/apify/crawlee-python/actions
Security and quality1 (1)https://github.com/apify/crawlee-python/security
Insightshttps://github.com/apify/crawlee-python/insights
https://github.com/apify/crawlee-python
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:throttling-sub-openerhttps://github.com/Mantisus/crawlee-python/tree/throttling-sub-opener
Conversationhttps://github.com/apify/crawlee-python/pull/2177
Commits11 (11)https://github.com/apify/crawlee-python/pull/2177/commits
Checkshttps://github.com/apify/crawlee-python/pull/2177/checks
Files changedhttps://github.com/apify/crawlee-python/pull/2177/files
fix(throttling-manager): Open the per-domain sub-queues on first use instead of on inserthttps://github.com/apify/crawlee-python/pull/2177#top
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:throttling-sub-openerhttps://github.com/Mantisus/crawlee-python/tree/throttling-sub-opener
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2177#issue-5197229628
ThrottlingRequestManager never reopens its sub-queues, so a restart strands throttled requests #2142https://github.com/apify/crawlee-python/issues/2142
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
https://github.com/Mantisus
open the per-domain sub-queues on first use instead of on inserthttps://github.com/apify/crawlee-python/pull/2177/commits/f25b9a8084659dcccdb17fbbceb06c08ebda9df0
f25b9a8https://github.com/apify/crawlee-python/pull/2177/commits/f25b9a8084659dcccdb17fbbceb06c08ebda9df0
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2177#event-29708967275
https://github.com/vdusek
vdusekhttps://github.com/vdusek
apify-factoryhttps://github.com/apify-factory
Copilothttps://github.com/apps/copilot-pull-request-reviewer
August 20, 2026 07:32https://github.com/apify/crawlee-python/pull/2177#event-29728750071
started reviewinghttps://github.com/apify/crawlee-python/sessions/6909d8d4-66a2-4714-9d9b-8a4e4f58b81f
vdusekhttps://github.com/vdusek
August 20, 2026 09:42https://github.com/apify/crawlee-python/pull/2177#event-29735097168
View session https://github.com/apify/crawlee-python/sessions/6909d8d4-66a2-4714-9d9b-8a4e4f58b81f
https://github.com/apps/copilot-pull-request-reviewer
Copilothttps://github.com/apps/copilot-pull-request-reviewer
Aug 20, 2026 https://github.com/apify/crawlee-python/pull/2177#pullrequestreview-4981322889
View reviewed changes https://github.com/apify/crawlee-python/pull/2177/files/f25b9a8084659dcccdb17fbbceb06c08ebda9df0
Copilothttps://github.com/apps/copilot-pull-request-reviewer
https://github.com/apify/crawlee-python/pull/2177#pullrequestreview-4981322889
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Add a code-review agent skillhttps://github.com/apify/crawlee-python/new/master?filename=.github/skills/code-review/SKILL.md
Learn more in the docs.https://docs.github.com/en/copilot/how-tos/use-copilot-agents/request-a-code-review/use-code-review#mcp-servers-and-agent-skills
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
src/crawlee/request_loaders/_throttling_request_manager.pyhttps://github.com/apify/crawlee-python/pull/2177/files/f25b9a8084659dcccdb17fbbceb06c08ebda9df0#diff-e7019a11ff787a5d190c7b3b77be9cd1ee48fb56b4c6e39018d956be91aa4f72
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
src/crawlee/request_loaders/_throttling_request_manager.pyhttps://github.com/apify/crawlee-python/pull/2177/files/f25b9a8084659dcccdb17fbbceb06c08ebda9df0#diff-e7019a11ff787a5d190c7b3b77be9cd1ee48fb56b4c6e39018d956be91aa4f72
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
src/crawlee/request_loaders/_throttling_request_manager.pyhttps://github.com/apify/crawlee-python/pull/2177/files/f25b9a8084659dcccdb17fbbceb06c08ebda9df0#diff-e7019a11ff787a5d190c7b3b77be9cd1ee48fb56b4c6e39018d956be91aa4f72
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
src/crawlee/request_loaders/_throttling_request_manager.pyhttps://github.com/apify/crawlee-python/pull/2177/files/f25b9a8084659dcccdb17fbbceb06c08ebda9df0#diff-e7019a11ff787a5d190c7b3b77be9cd1ee48fb56b4c6e39018d956be91aa4f72
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
vdusekhttps://github.com/vdusek
August 20, 2026 14:02https://github.com/apify/crawlee-python/pull/2177#commits-pushed-3a389f8
https://github.com/vdusek
fix(throttling-manager): key in-flight inner requests by unique key a…https://github.com/apify/crawlee-python/pull/2177/commits/3a389f824c861fa10582e6836a044a07383d98f2
3a389f8https://github.com/apify/crawlee-python/pull/2177/commits/3a389f824c861fa10582e6836a044a07383d98f2
https://github.com/vdusek
refactor(throttling-manager): inline _select_manager into its only ca…https://github.com/apify/crawlee-python/pull/2177/commits/873c127176b06e4bd082936b0426b147d520223d
873c127https://github.com/apify/crawlee-python/pull/2177/commits/873c127176b06e4bd082936b0426b147d520223d
https://github.com/vdusek
test(throttling-manager): cover the default purge-on-start path for s…https://github.com/apify/crawlee-python/pull/2177/commits/2dee3645f756415084f61a3c3deddad0f991d27f
2dee364https://github.com/apify/crawlee-python/pull/2177/commits/2dee3645f756415084f61a3c3deddad0f991d27f
https://github.com/vdusek
docs(throttling-manager): document sub-manager storage and purge-on-s…https://github.com/apify/crawlee-python/pull/2177/commits/986fd269f65fc9652e12d022ac604bb27c2aaa29
986fd26https://github.com/apify/crawlee-python/pull/2177/commits/986fd269f65fc9652e12d022ac604bb27c2aaa29
https://github.com/vdusek
fix(throttling-manager): keep the in-flight record until the completi…https://github.com/apify/crawlee-python/pull/2177/commits/898e55c556af2df422961164972fc716a86b128b
898e55chttps://github.com/apify/crawlee-python/pull/2177/commits/898e55c556af2df422961164972fc716a86b128b
https://github.com/vdusek
fix(throttling-manager): keep sub-managers that opened before a sibli…https://github.com/apify/crawlee-python/pull/2177/commits/18825311ce682ad0aeea81505314b59578cf9bff
1882531https://github.com/apify/crawlee-python/pull/2177/commits/18825311ce682ad0aeea81505314b59578cf9bff
https://github.com/vdusek
docs(throttling-manager): clarify when sub-managers open and how in-f…https://github.com/apify/crawlee-python/pull/2177/commits/3fff872db05856da93911975378be947d8291f75
3fff872https://github.com/apify/crawlee-python/pull/2177/commits/3fff872db05856da93911975378be947d8291f75
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Aug 20, 2026 https://github.com/apify/crawlee-python/pull/2177#pullrequestreview-4982672998
View reviewed changes https://github.com/apify/crawlee-python/pull/2177/files/3fff872db05856da93911975378be947d8291f75
vdusekhttps://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2177#pullrequestreview-4982672998
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
https://github.com/vdusek
docs(throttling-manager): tighten the comments and docstringshttps://github.com/apify/crawlee-python/pull/2177/commits/255ed412983dff8557ac12137751622dc7d479b3
255ed41https://github.com/apify/crawlee-python/pull/2177/commits/255ed412983dff8557ac12137751622dc7d479b3
https://github.com/vdusek
vdusekhttps://github.com/vdusek
apify-factoryhttps://github.com/apify-factory
August 21, 2026 06:04https://github.com/apify/crawlee-python/pull/2177#event-29786034538
vdusekhttps://github.com/vdusek
August 21, 2026 08:31https://github.com/apify/crawlee-python/pull/2177#commits-pushed-98c6ef5
https://github.com/vdusek
Merge branch 'master' into throttling-sub-openerhttps://github.com/apify/crawlee-python/pull/2177/commits/98c6ef5a3463ce98c86da36a77fc064f0f3a4e7a
98c6ef5https://github.com/apify/crawlee-python/pull/2177/commits/98c6ef5a3463ce98c86da36a77fc064f0f3a4e7a
https://github.com/vdusek
docs: correct the deduplication claim in the request throttling guidehttps://github.com/apify/crawlee-python/pull/2177/commits/23b762af3ed61ba5727b969d7791c9d0b2bcac3e
23b762ahttps://github.com/apify/crawlee-python/pull/2177/commits/23b762af3ed61ba5727b969d7791c9d0b2bcac3e
https://github.com/vdusek
vdusekhttps://github.com/vdusek
991502fhttps://github.com/apify/crawlee-python/commit/991502f61e6313d22ccdf140f0cdb012287b53f3
Aug 21, 2026https://github.com/apify/crawlee-python/pull/2177#event-29787435943
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
https://github.co/hiddenchars
https://github.com/apify/crawlee-python/pull/{{ revealButtonHref }}
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2177
Copilot code review https://github.com/apps/copilot-pull-request-reviewer
Copilot https://github.com/apps/copilot-pull-request-reviewer
Learn more about Copilot code review.https://gh.io/copilot-code-review-docs
https://github.com/apify/crawlee-python/pull/2177/files/f25b9a8084659dcccdb17fbbceb06c08ebda9df0
https://github.com/vdusek
vdusek https://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2177/files/3fff872db05856da93911975378be947d8291f75
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
https://github.com/Mantisus
Mantisus https://github.com/Mantisus
ThrottlingRequestManager never reopens its sub-queues, so a restart strands throttled requests https://github.com/apify/crawlee-python/issues/2142
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2177
https://github.com/Mantisus
https://github.com/vdusek
https://github.com/apps/copilot-pull-request-reviewer
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.