René's URL Explorer Experiment


Title: fix(throttling-manager): Stop holding a concurrency slot while a domain is throttled by Mantisus · Pull Request #2176 · apify/crawlee-python · GitHub

Open Graph Title: fix(throttling-manager): Stop holding a concurrency slot while a domain is throttled by Mantisus · Pull Request #2176 · apify/crawlee-python

X Title: fix(throttling-manager): Stop holding a concurrency slot while a domain is throttled by Mantisus · Pull Request #2176 · apify/crawlee-python

Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included. - fix(throttling-manager): Stop holding a concurrency slot while a domain is throttled by Mantisus · Pull Request #2176 · apify/crawlee-python

Open Graph Description: Description fetch_next_request now returns None instead of sleeping out a cooldown, and is_empty() counts only domains that are not throttled, so the autoscaled pool idles instead of holding worke...

X Description: Description fetch_next_request now returns None instead of sleeping out a cooldown, and is_empty() counts only domains that are not throttled, so the autoscaled pool idles instead of holding worke...

Opengraph URL: https://github.com/apify/crawlee-python/pull/2176

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:49b4057e-b708-04d3-c54f-598622752e98
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD554:6FAD8:BD80EF:F4F3DF:6B00EBB3
html-safe-nonce84b5c3a57fbd6d491610f988104dc9e4cd559c19ab2a3bcf869e417f4817514c
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTU0OjZGQUQ4OkJEODBFRjpGNEYzREY6NkIwMEVCQjMiLCJ2aXNpdG9yX2lkIjoiMTQ0MzgzMzYwMDAwMzg2MTQyNyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac40a8185ec4ba8747b544524d3f97836a7adae13480825ead268c953eb513e810
hovercard-subject-tagpull_request:4316418976
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/apify/crawlee-python/2176/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/86b4a091e3de17382fa279e24fc4d13b6ee542889e130533f08116d72b874c04/apify/crawlee-python/pull/2176
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/86b4a091e3de17382fa279e24fc4d13b6ee542889e130533f08116d72b874c04/apify/crawlee-python/pull/2176
og:image:altDescription fetch_next_request now returns None instead of sleeping out a cooldown, and is_empty() counts only domains that are not throttled, so the autoscaled pool idles instead of holding worke...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameMantisus
hostnamegithub.com
expected-hostnamegithub.com
None4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc
turbo-cache-controlno-cache
go-importgithub.com/apify/crawlee-python git https://github.com/apify/crawlee-python.git
octolytics-dimension-user_id24586296
octolytics-dimension-user_loginapify
octolytics-dimension-repository_id741365632
octolytics-dimension-repository_nwoapify/crawlee-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id741365632
octolytics-dimension-repository_network_root_nwoapify/crawlee-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release9983dd5da06943f833c971dd5b6bc988e2858be5
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-namepull-requests

Links:

Skip to contenthttps://github.com/apify/crawlee-python/pull/2176#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2176
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2176
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=apify%2Fcrawlee-python
Reloadhttps://github.com/apify/crawlee-python/pull/2176
Reloadhttps://github.com/apify/crawlee-python/pull/2176
Reloadhttps://github.com/apify/crawlee-python/pull/2176
apifyhttps://github.com/apify
crawlee-pythonhttps://github.com/apify/crawlee-python
Codehttps://github.com/apify/crawlee-python
Issues72 (72)https://github.com/apify/crawlee-python/issues
Pull requests14 (14)https://github.com/apify/crawlee-python/pulls
Discussionshttps://github.com/apify/crawlee-python/discussions
Actionshttps://github.com/apify/crawlee-python/actions
Security and quality1 (1)https://github.com/apify/crawlee-python/security
Insightshttps://github.com/apify/crawlee-python/insights
https://github.com/apify/crawlee-python
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:throttling-fetch-requesthttps://github.com/Mantisus/crawlee-python/tree/throttling-fetch-request
Conversationhttps://github.com/apify/crawlee-python/pull/2176
Commits5 (5)https://github.com/apify/crawlee-python/pull/2176/commits
Checkshttps://github.com/apify/crawlee-python/pull/2176/checks
Files changedhttps://github.com/apify/crawlee-python/pull/2176/files
fix(throttling-manager): Stop holding a concurrency slot while a domain is throttledhttps://github.com/apify/crawlee-python/pull/2176#top
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:throttling-fetch-requesthttps://github.com/Mantisus/crawlee-python/tree/throttling-fetch-request
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2176#issue-5195798824
ThrottlingRequestManager.fetch_next_request holds a concurrency slot while waiting out a throttle #2127https://github.com/apify/crawlee-python/issues/2127
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2176
https://github.com/Mantisus
stop holding a concurrency slot while a domain is throttledhttps://github.com/apify/crawlee-python/pull/2176/commits/62439854496083460d5b2e9ac3d21a278165e85c
6243985https://github.com/apify/crawlee-python/pull/2176/commits/62439854496083460d5b2e9ac3d21a278165e85c
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2176#event-29700146753
https://github.com/vdusek
vdusekhttps://github.com/vdusek
apify-factoryhttps://github.com/apify-factory
Copilothttps://github.com/apps/copilot-pull-request-reviewer
August 20, 2026 07:32https://github.com/apify/crawlee-python/pull/2176#event-29728774928
started reviewinghttps://github.com/apify/crawlee-python/sessions/485fdd93-3189-4dff-84b8-3f72eb96be63
vdusekhttps://github.com/vdusek
August 20, 2026 09:42https://github.com/apify/crawlee-python/pull/2176#event-29735096712
View session https://github.com/apify/crawlee-python/sessions/485fdd93-3189-4dff-84b8-3f72eb96be63
https://github.com/apps/copilot-pull-request-reviewer
Copilothttps://github.com/apps/copilot-pull-request-reviewer
Aug 20, 2026 https://github.com/apify/crawlee-python/pull/2176#pullrequestreview-4981314376
View reviewed changes https://github.com/apify/crawlee-python/pull/2176/files/62439854496083460d5b2e9ac3d21a278165e85c
Copilothttps://github.com/apps/copilot-pull-request-reviewer
https://github.com/apify/crawlee-python/pull/2176#pullrequestreview-4981314376
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Add a code-review agent skillhttps://github.com/apify/crawlee-python/new/master?filename=.github/skills/code-review/SKILL.md
Learn more in the docs.https://docs.github.com/en/copilot/how-tos/use-copilot-agents/request-a-code-review/use-code-review#mcp-servers-and-agent-skills
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2176
vdusekhttps://github.com/vdusek
August 20, 2026 13:59https://github.com/apify/crawlee-python/pull/2176#commits-pushed-f4d84c5
https://github.com/vdusek
docs: describe concurrency slot release in the request throttling guidehttps://github.com/apify/crawlee-python/pull/2176/commits/f4d84c557ba22d9c9a19bcd49bcafc4f01e29aab
f4d84c5https://github.com/apify/crawlee-python/pull/2176/commits/f4d84c557ba22d9c9a19bcd49bcafc4f01e29aab
https://github.com/vdusek
refactor(throttling): clarify the is_empty and is_finished contracthttps://github.com/apify/crawlee-python/pull/2176/commits/020b386b3a0e4ff1e50fad5e713230b049288105
020b386https://github.com/apify/crawlee-python/pull/2176/commits/020b386b3a0e4ff1e50fad5e713230b049288105
https://github.com/vdusek
test(throttling): cover a throttled crawl finishing without ending earlyhttps://github.com/apify/crawlee-python/pull/2176/commits/8840a33e8513a6ebcf1be5d2879f173b501d705a
8840a33https://github.com/apify/crawlee-python/pull/2176/commits/8840a33e8513a6ebcf1be5d2879f173b501d705a
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Aug 20, 2026 https://github.com/apify/crawlee-python/pull/2176#pullrequestreview-4982457323
View reviewed changes https://github.com/apify/crawlee-python/pull/2176/files/8840a33e8513a6ebcf1be5d2879f173b501d705a
vdusekhttps://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2176#pullrequestreview-4982457323
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2176
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Pijukatelhttps://github.com/Pijukatel
apify-factoryhttps://github.com/apify-factory
August 20, 2026 12:03https://github.com/apify/crawlee-python/pull/2176#event-29741657617
https://github.com/vdusek
Merge branch 'master' into throttling-fetch-requesthttps://github.com/apify/crawlee-python/pull/2176/commits/101f9441b3eac52f755d50679475d0963f16d892
101f944https://github.com/apify/crawlee-python/pull/2176/commits/101f9441b3eac52f755d50679475d0963f16d892
https://github.com/vdusek
vdusekhttps://github.com/vdusek
5c1fa09https://github.com/apify/crawlee-python/commit/5c1fa09cc36e2a3c042740767cfc974743267ee3
Aug 21, 2026https://github.com/apify/crawlee-python/pull/2176#event-29786875551
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2176
https://github.co/hiddenchars
https://github.com/apify/crawlee-python/pull/{{ revealButtonHref }}
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2176
Copilot code review https://github.com/apps/copilot-pull-request-reviewer
Copilot https://github.com/apps/copilot-pull-request-reviewer
Learn more about Copilot code review.https://gh.io/copilot-code-review-docs
https://github.com/apify/crawlee-python/pull/2176/files/62439854496083460d5b2e9ac3d21a278165e85c
https://github.com/vdusek
vdusek https://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2176/files/8840a33e8513a6ebcf1be5d2879f173b501d705a
https://github.com/Pijukatel
Pijukatel https://github.com/Pijukatel
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2176
https://github.com/Mantisus
Mantisus https://github.com/Mantisus
ThrottlingRequestManager.fetch_next_request holds a concurrency slot while waiting out a throttle https://github.com/apify/crawlee-python/issues/2127
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2176
https://github.com/Mantisus
https://github.com/vdusek
https://github.com/apps/copilot-pull-request-reviewer
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.