René's URL Explorer Experiment


Title: fix(throttling-manager): Apply per-domain backoff when `retry_on_blocked` is disabled by Mantisus · Pull Request #2158 · apify/crawlee-python · GitHub

Open Graph Title: fix(throttling-manager): Apply per-domain backoff when `retry_on_blocked` is disabled by Mantisus · Pull Request #2158 · apify/crawlee-python

X Title: fix(throttling-manager): Apply per-domain backoff when `retry_on_blocked` is disabled by Mantisus · Pull Request #2158 · apify/crawlee-python

Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included. - fix(throttling-manager): Apply per-domain backoff when `retry_on_blocked` is disabled by Mantisus · Pull Request #2158 · apify/crawlee-python

Open Graph Description: Description Recording a 429 is now separate from the session blocking check and runs regardless of retry_on_blocked. That configuration previously dropped the 429 backoff and the robots.txt crawl-...

X Description: Description Recording a 429 is now separate from the session blocking check and runs regardless of retry_on_blocked. That configuration previously dropped the 429 backoff and the robots.txt crawl-...

Opengraph URL: https://github.com/apify/crawlee-python/pull/2158

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:c8bced98-ffc6-4017-05d2-ae0e9b772909
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idBAE4:CC1A4:A1616B:CED2D5:6B00F197
html-safe-nonce28325090ab05d8c4448e1b5bb34c12401157bdb80c2d6078cec3820fea621caa
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCQUU0OkNDMUE0OkExNjE2QjpDRUQyRDU6NkIwMEYxOTciLCJ2aXNpdG9yX2lkIjoiMTQxNDEzODA2NzI3MDMwMDA1NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac56eb1bac5dbed5f99dbfee7f88e3a22b58715b176579b9e5e31bb94286d62d02
hovercard-subject-tagpull_request:4281978909
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/apify/crawlee-python/2158/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/4d6bdb5ab5b8ce4269cac169622028a391b15e9638ae28eee49682ce1d36a404/apify/crawlee-python/pull/2158
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/4d6bdb5ab5b8ce4269cac169622028a391b15e9638ae28eee49682ce1d36a404/apify/crawlee-python/pull/2158
og:image:altDescription Recording a 429 is now separate from the session blocking check and runs regardless of retry_on_blocked. That configuration previously dropped the 429 backoff and the robots.txt crawl-...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameMantisus
hostnamegithub.com
expected-hostnamegithub.com
None4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc
turbo-cache-controlno-cache
go-importgithub.com/apify/crawlee-python git https://github.com/apify/crawlee-python.git
octolytics-dimension-user_id24586296
octolytics-dimension-user_loginapify
octolytics-dimension-repository_id741365632
octolytics-dimension-repository_nwoapify/crawlee-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id741365632
octolytics-dimension-repository_network_root_nwoapify/crawlee-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release9983dd5da06943f833c971dd5b6bc988e2858be5
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-namepull-requests

Links:

Skip to contenthttps://github.com/apify/crawlee-python/pull/2158#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2158
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2158
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=apify%2Fcrawlee-python
Reloadhttps://github.com/apify/crawlee-python/pull/2158
Reloadhttps://github.com/apify/crawlee-python/pull/2158
Reloadhttps://github.com/apify/crawlee-python/pull/2158
apifyhttps://github.com/apify
crawlee-pythonhttps://github.com/apify/crawlee-python
Codehttps://github.com/apify/crawlee-python
Issues72 (72)https://github.com/apify/crawlee-python/issues
Pull requests14 (14)https://github.com/apify/crawlee-python/pulls
Discussionshttps://github.com/apify/crawlee-python/discussions
Actionshttps://github.com/apify/crawlee-python/actions
Security and quality1 (1)https://github.com/apify/crawlee-python/security
Insightshttps://github.com/apify/crawlee-python/insights
https://github.com/apify/crawlee-python
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:tandem-throttlinghttps://github.com/Mantisus/crawlee-python/tree/tandem-throttling
Conversationhttps://github.com/apify/crawlee-python/pull/2158
Commits7 (7)https://github.com/apify/crawlee-python/pull/2158/commits
Checkshttps://github.com/apify/crawlee-python/pull/2158/checks
Files changedhttps://github.com/apify/crawlee-python/pull/2158/files
fix(throttling-manager): Apply per-domain backoff when retry_on_blocked is disabledhttps://github.com/apify/crawlee-python/pull/2158#top
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:tandem-throttlinghttps://github.com/Mantisus/crawlee-python/tree/tandem-throttling
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 14, 2026https://github.com/apify/crawlee-python/pull/2158#issue-5154230377
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
ThrottlingRequestManager silently does nothing in two configurations #2126https://github.com/apify/crawlee-python/issues/2126
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
janbucharhttps://github.com/janbuchar
August 14, 2026 17:38https://github.com/apify/crawlee-python/pull/2158#event-29472538109
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 14, 2026https://github.com/apify/crawlee-python/pull/2158#event-29472542304
https://github.com/janbuchar
janbucharhttps://github.com/janbuchar
Aug 18, 2026https://github.com/apify/crawlee-python/pull/2158#issuecomment-5328334859
@Mantisushttps://github.com/Mantisus
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After #2159 https://github.com/apify/crawlee-python/pull/2159
fix(throttling-manager): Normalize configured domains the same way as crawled hostnames #2157 https://github.com/apify/crawlee-python/pull/2157
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2158#event-29708490511
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
force-pushedhttps://github.com/apify/crawlee-python/compare/c7415a68585fa1d7139b1e22493e24e65e0db289..9f8aaf91614071f14618b6065c97a0889887b1eb
c7415a6https://github.com/apify/crawlee-python/commit/c7415a68585fa1d7139b1e22493e24e65e0db289
9f8aaf9https://github.com/apify/crawlee-python/commit/9f8aaf91614071f14618b6065c97a0889887b1eb
Compare https://github.com/apify/crawlee-python/compare/c7415a68585fa1d7139b1e22493e24e65e0db289..9f8aaf91614071f14618b6065c97a0889887b1eb
August 21, 2026 21:28https://github.com/apify/crawlee-python/pull/2158#event-29829078016
https://github.com/janbuchar
janbucharhttps://github.com/janbuchar
Aug 26, 2026https://github.com/apify/crawlee-python/pull/2158#issuecomment-5430868676
@Mantisushttps://github.com/Mantisus
apify/crawlee#4061https://github.com/apify/crawlee/pull/4061
@vdusekhttps://github.com/vdusek
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Aug 27, 2026https://github.com/apify/crawlee-python/pull/2158#issuecomment-5435324565
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
@Mantisushttps://github.com/Mantisus
apify/crawlee#4061https://github.com/apify/crawlee/pull/4061
@vdusekhttps://github.com/vdusek
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/vdusek
vdusekhttps://github.com/vdusek
August 31, 2026 07:11https://github.com/apify/crawlee-python/pull/2158#event-30263029312
https://github.com/Mantisus
apply throttling behind a tandem and with retry_on_blocked disabledhttps://github.com/apify/crawlee-python/pull/2158/commits/dc1f99481a38762049f29d511cf0ec8cd9651ad3
dc1f994https://github.com/apify/crawlee-python/pull/2158/commits/dc1f99481a38762049f29d511cf0ec8cd9651ad3
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
force-pushedhttps://github.com/apify/crawlee-python/compare/9f8aaf91614071f14618b6065c97a0889887b1eb..dc1f99481a38762049f29d511cf0ec8cd9651ad3
9f8aaf9https://github.com/apify/crawlee-python/commit/9f8aaf91614071f14618b6065c97a0889887b1eb
dc1f994https://github.com/apify/crawlee-python/commit/dc1f99481a38762049f29d511cf0ec8cd9651ad3
Compare https://github.com/apify/crawlee-python/compare/9f8aaf91614071f14618b6065c97a0889887b1eb..dc1f99481a38762049f29d511cf0ec8cd9651ad3
September 3, 2026 21:00https://github.com/apify/crawlee-python/pull/2158#event-30516149627
https://github.com/Mantisus
remove changes to the request loaderhttps://github.com/apify/crawlee-python/pull/2158/commits/496eedbf88b56a541bfcc1f0ec6fb9b05cd01955
496eedbhttps://github.com/apify/crawlee-python/pull/2158/commits/496eedbf88b56a541bfcc1f0ec6fb9b05cd01955
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Sep 3, 2026https://github.com/apify/crawlee-python/pull/2158#event-30516272024
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
September 3, 2026 21:04https://github.com/apify/crawlee-python/pull/2158#event-30516357989
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Sep 3, 2026https://github.com/apify/crawlee-python/pull/2158#issuecomment-5532140089
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
vdusekhttps://github.com/vdusek
September 3, 2026 21:06https://github.com/apify/crawlee-python/pull/2158#event-30516461587
vdusekhttps://github.com/vdusek
September 10, 2026 08:48https://github.com/apify/crawlee-python/pull/2158#commits-pushed-fa57661
https://github.com/vdusek
docs: Describe 429 recording in the status-code handler docstringshttps://github.com/apify/crawlee-python/pull/2158/commits/fa57661c132a217d710170becb3ed95b116a7cd9
fa57661https://github.com/apify/crawlee-python/pull/2158/commits/fa57661c132a217d710170becb3ed95b116a7cd9
https://github.com/vdusek
test: Tidy the queue alias and retry comment in the 429 recording testhttps://github.com/apify/crawlee-python/pull/2158/commits/3c74823dc48d24cc7c29f0baf255ab2979465024
3c74823https://github.com/apify/crawlee-python/pull/2158/commits/3c74823dc48d24cc7c29f0baf255ab2979465024
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 10, 2026https://github.com/apify/crawlee-python/pull/2158#event-30881780565
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 10, 2026 https://github.com/apify/crawlee-python/pull/2158#pullrequestreview-5163632675
View reviewed changes https://github.com/apify/crawlee-python/pull/2158/files/3c74823dc48d24cc7c29f0baf255ab2979465024
vdusekhttps://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2158#pullrequestreview-5163632675
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 10, 2026https://github.com/apify/crawlee-python/pull/2158#issuecomment-5620505372
@janbucharhttps://github.com/janbuchar
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
vdusekhttps://github.com/vdusek
September 16, 2026 10:53https://github.com/apify/crawlee-python/pull/2158#commits-pushed-36e0fb6
https://github.com/vdusek
docs: Drop the restated sentence from the 429 recording docstringhttps://github.com/apify/crawlee-python/pull/2158/commits/36e0fb686c77f288545d63432fecd8277cedf5f9
36e0fb6https://github.com/apify/crawlee-python/pull/2158/commits/36e0fb686c77f288545d63432fecd8277cedf5f9
https://github.com/vdusek
test: Cover the session-blocked path in the 429 recording testhttps://github.com/apify/crawlee-python/pull/2158/commits/bcabbc3125048f3680e9f1a9a3907a10b2f570d5
bcabbc3https://github.com/apify/crawlee-python/pull/2158/commits/bcabbc3125048f3680e9f1a9a3907a10b2f570d5
https://github.com/vdusek
test: Stop the 429 recording assertion racing the backoff expiryhttps://github.com/apify/crawlee-python/pull/2158/commits/11c8cd0c10f56950fa30d6cd32c2858325ce1b3b
11c8cd0https://github.com/apify/crawlee-python/pull/2158/commits/11c8cd0c10f56950fa30d6cd32c2858325ce1b3b
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 16, 2026 https://github.com/apify/crawlee-python/pull/2158#ref-issue-5472505383
ThrottlingRequestManager has no effect behind RequestManagerTandem #2238 https://github.com/apify/crawlee-python/issues/2238
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 16, 2026https://github.com/apify/crawlee-python/pull/2158#issuecomment-5694977125
#2126https://github.com/apify/crawlee-python/issues/2126
#2238https://github.com/apify/crawlee-python/issues/2238
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/vdusek
vdusekhttps://github.com/vdusek
9c23101https://github.com/apify/crawlee-python/commit/9c2310180478c2821d006e514ec55bfbfec4b4dc
Sep 16, 2026https://github.com/apify/crawlee-python/pull/2158#event-31233185390
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.co/hiddenchars
https://github.com/apify/crawlee-python/pull/{{ revealButtonHref }}
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2158
https://github.com/vdusek
vdusek https://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2158/files/3c74823dc48d24cc7c29f0baf255ab2979465024
https://github.com/janbuchar
janbuchar https://github.com/janbuchar
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/Mantisus
Mantisus https://github.com/Mantisus
ThrottlingRequestManager silently does nothing in two configurations https://github.com/apify/crawlee-python/issues/2126
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2158
https://github.com/Mantisus
https://github.com/janbuchar
https://github.com/vdusek
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.