René's URL Explorer Experiment


Title: fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After by Mantisus · Pull Request #2159 · apify/crawlee-python · GitHub

Open Graph Title: fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After by Mantisus · Pull Request #2159 · apify/crawlee-python

X Title: fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After by Mantisus · Pull Request #2159 · apify/crawlee-python

Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included. - fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-After by Mantisus · Pull Request #2159 · apify/crawlee-python

Open Graph Description: Description Only the first 429 of a burst now advances the exponent, and it decays once the domain has left us alone for a full extra backoff window. Previously every concurrent 429 stepped it, so...

X Description: Description Only the first 429 of a burst now advances the exponent, and it decays once the domain has left us alone for a full extra backoff window. Previously every concurrent 429 stepped it, so...

Opengraph URL: https://github.com/apify/crawlee-python/pull/2159

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:6c5f3bef-9ee2-3ed2-7123-2a0740bb5f47
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idE3C8:14F601:C4E010:FD7797:6B00EB93
html-safe-noncea7b72db55ddd63c94881408788a76bc9b4968ad479dded48fdc0b14d6a540adb
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFM0M4OjE0RjYwMTpDNEUwMTA6RkQ3Nzk3OjZCMDBFQjkzIiwidmlzaXRvcl9pZCI6Ijc0MzgwNzY3NTUwMTkwOTkwMjciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacafb5f0783972c738f50135632b0b692a287ff2a452e5fd1bc8fab8ad6bc451a2
hovercard-subject-tagpull_request:4282006191
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/apify/crawlee-python/2159/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/22bff840784b3cbf063fa831e47ca0e240eda0b864c42871345cf43f7715fad3/apify/crawlee-python/pull/2159
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/22bff840784b3cbf063fa831e47ca0e240eda0b864c42871345cf43f7715fad3/apify/crawlee-python/pull/2159
og:image:altDescription Only the first 429 of a burst now advances the exponent, and it decays once the domain has left us alone for a full extra backoff window. Previously every concurrent 429 stepped it, so...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameMantisus
hostnamegithub.com
expected-hostnamegithub.com
None4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc
turbo-cache-controlno-cache
go-importgithub.com/apify/crawlee-python git https://github.com/apify/crawlee-python.git
octolytics-dimension-user_id24586296
octolytics-dimension-user_loginapify
octolytics-dimension-repository_id741365632
octolytics-dimension-repository_nwoapify/crawlee-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id741365632
octolytics-dimension-repository_network_root_nwoapify/crawlee-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release9983dd5da06943f833c971dd5b6bc988e2858be5
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-namepull-requests

Links:

Skip to contenthttps://github.com/apify/crawlee-python/pull/2159#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2159
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2159
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=apify%2Fcrawlee-python
Reloadhttps://github.com/apify/crawlee-python/pull/2159
Reloadhttps://github.com/apify/crawlee-python/pull/2159
Reloadhttps://github.com/apify/crawlee-python/pull/2159
apifyhttps://github.com/apify
crawlee-pythonhttps://github.com/apify/crawlee-python
Codehttps://github.com/apify/crawlee-python
Issues72 (72)https://github.com/apify/crawlee-python/issues
Pull requests14 (14)https://github.com/apify/crawlee-python/pulls
Discussionshttps://github.com/apify/crawlee-python/discussions
Actionshttps://github.com/apify/crawlee-python/actions
Security and quality1 (1)https://github.com/apify/crawlee-python/security
Insightshttps://github.com/apify/crawlee-python/insights
https://github.com/apify/crawlee-python
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:fix-backoff-429https://github.com/Mantisus/crawlee-python/tree/fix-backoff-429
Conversationhttps://github.com/apify/crawlee-python/pull/2159
Commits8 (8)https://github.com/apify/crawlee-python/pull/2159/commits
Checkshttps://github.com/apify/crawlee-python/pull/2159/checks
Files changedhttps://github.com/apify/crawlee-python/pull/2159/files
fix(throttling-manager): Correct the backoff for concurrent 429s and zero Retry-Afterhttps://github.com/apify/crawlee-python/pull/2159#top
vdusekhttps://github.com/vdusek
apify:masterhttps://github.com/apify/crawlee-python/tree/master
Mantisus:fix-backoff-429https://github.com/Mantisus/crawlee-python/tree/fix-backoff-429
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 14, 2026https://github.com/apify/crawlee-python/pull/2159#issue-5154263816
ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0 #2125https://github.com/apify/crawlee-python/issues/2125
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.com/Mantisus
correct the backoff for concurrent 429s and zero Retry-Afterhttps://github.com/apify/crawlee-python/pull/2159/commits/4b63cfe3017a877b71a4e60338beddb90cdccc65
4b63cfehttps://github.com/apify/crawlee-python/pull/2159/commits/4b63cfe3017a877b71a4e60338beddb90cdccc65
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
Aug 14, 2026https://github.com/apify/crawlee-python/pull/2159#event-29472744546
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2159#issuecomment-5341000420
@janbucharhttps://github.com/janbuchar
#2158 (comment)https://github.com/apify/crawlee-python/pull/2158#issuecomment-5328334859
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.com/Mantisus
Mantisushttps://github.com/Mantisus
janbucharhttps://github.com/janbuchar
August 19, 2026 10:55https://github.com/apify/crawlee-python/pull/2159#event-29676963460
https://github.com/janbuchar
janbucharhttps://github.com/janbuchar
Aug 19, 2026https://github.com/apify/crawlee-python/pull/2159#issuecomment-5343730282
@vdusekhttps://github.com/vdusek
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.com/vdusek
vdusekhttps://github.com/vdusek
apify-factoryhttps://github.com/apify-factory
Copilothttps://github.com/apps/copilot-pull-request-reviewer
August 20, 2026 09:41https://github.com/apify/crawlee-python/pull/2159#event-29735048778
started reviewinghttps://github.com/apify/crawlee-python/sessions/749d990e-a688-4d3a-8c32-745dd6129afd
vdusekhttps://github.com/vdusek
August 20, 2026 09:42https://github.com/apify/crawlee-python/pull/2159#event-29735094498
View session https://github.com/apify/crawlee-python/sessions/749d990e-a688-4d3a-8c32-745dd6129afd
https://github.com/apps/copilot-pull-request-reviewer
Copilothttps://github.com/apps/copilot-pull-request-reviewer
Aug 20, 2026 https://github.com/apify/crawlee-python/pull/2159#pullrequestreview-4981325786
View reviewed changes https://github.com/apify/crawlee-python/pull/2159/files/4b63cfe3017a877b71a4e60338beddb90cdccc65
Copilothttps://github.com/apps/copilot-pull-request-reviewer
https://github.com/apify/crawlee-python/pull/2159#pullrequestreview-4981325786
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Add a code-review agent skillhttps://github.com/apify/crawlee-python/new/master?filename=.github/skills/code-review/SKILL.md
Learn more in the docs.https://docs.github.com/en/copilot/how-tos/use-copilot-agents/request-a-code-review/use-code-review#mcp-servers-and-agent-skills
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
src/crawlee/request_loaders/_throttling_request_manager.pyhttps://github.com/apify/crawlee-python/pull/2159/files/4b63cfe3017a877b71a4e60338beddb90cdccc65#diff-e7019a11ff787a5d190c7b3b77be9cd1ee48fb56b4c6e39018d956be91aa4f72
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
vdusekhttps://github.com/vdusek
August 20, 2026 14:01https://github.com/apify/crawlee-python/pull/2159#commits-pushed-f779623
https://github.com/vdusek
fix(throttling-manager): Escalate the 429 backoff when a crawl-delay …https://github.com/apify/crawlee-python/pull/2159/commits/f779623812ce002700a4ba68db51e56082acf310
f779623https://github.com/apify/crawlee-python/pull/2159/commits/f779623812ce002700a4ba68db51e56082acf310
https://github.com/vdusek
fix(throttling-manager): Clamp the 429 backoff exponent to keep the d…https://github.com/apify/crawlee-python/pull/2159/commits/fcf99824b9faafc36544895c6ddd8af0baf7f9c4
fcf9982https://github.com/apify/crawlee-python/pull/2159/commits/fcf99824b9faafc36544895c6ddd8af0baf7f9c4
https://github.com/vdusek
test(throttling-manager): Assert a dispatch cannot shorten an active …https://github.com/apify/crawlee-python/pull/2159/commits/c477964b69b9176df0d56e28d9b4ca572052216d
c477964https://github.com/apify/crawlee-python/pull/2159/commits/c477964b69b9176df0d56e28d9b4ca572052216d
https://github.com/vdusek
chore(throttling-manager): Add a debug log for 429s ignored during an…https://github.com/apify/crawlee-python/pull/2159/commits/aa207da27eccbdf66e337eb1bd6fbc8d9fc08181
aa207dahttps://github.com/apify/crawlee-python/pull/2159/commits/aa207da27eccbdf66e337eb1bd6fbc8d9fc08181
https://github.com/vdusek
docs(throttling-manager): Clarify the 429 backoff window and Retry-Af…https://github.com/apify/crawlee-python/pull/2159/commits/4fdc07b884819f4a3d8a56405c79c0aac03969da
4fdc07bhttps://github.com/apify/crawlee-python/pull/2159/commits/4fdc07b884819f4a3d8a56405c79c0aac03969da
https://github.com/vdusek
chore(throttling-manager): Tidy up test helper names and drop an unus…https://github.com/apify/crawlee-python/pull/2159/commits/3d407e4827412a686fe6d7d6cef14ae639ce8b64
3d407e4https://github.com/apify/crawlee-python/pull/2159/commits/3d407e4827412a686fe6d7d6cef14ae639ce8b64
https://github.com/vdusek
vdusekhttps://github.com/vdusek
apify-factoryhttps://github.com/apify-factory
August 21, 2026 06:03https://github.com/apify/crawlee-python/pull/2159#event-29786022164
https://github.com/vdusek
Merge remote-tracking branch 'origin/master' into pr-2159https://github.com/apify/crawlee-python/pull/2159/commits/f969b41bb44017d9e510ffd64e5d185f447f4602
f969b41https://github.com/apify/crawlee-python/pull/2159/commits/f969b41bb44017d9e510ffd64e5d185f447f4602
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Aug 21, 2026 https://github.com/apify/crawlee-python/pull/2159#pullrequestreview-4990576514
View reviewed changes https://github.com/apify/crawlee-python/pull/2159/files/f969b41bb44017d9e510ffd64e5d185f447f4602
vdusekhttps://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2159#pullrequestreview-4990576514
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.com/vdusek
vdusekhttps://github.com/vdusek
142ec58https://github.com/apify/crawlee-python/commit/142ec58855c1823968d9638938eb6ef545db107f
Aug 31, 2026https://github.com/apify/crawlee-python/pull/2159#event-30263011428
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.co/hiddenchars
https://github.com/apify/crawlee-python/pull/{{ revealButtonHref }}
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2159
Copilot code review https://github.com/apps/copilot-pull-request-reviewer
Copilot https://github.com/apps/copilot-pull-request-reviewer
Learn more about Copilot code review.https://gh.io/copilot-code-review-docs
https://github.com/apify/crawlee-python/pull/2159/files/4b63cfe3017a877b71a4e60338beddb90cdccc65
https://github.com/vdusek
vdusek https://github.com/vdusek
https://github.com/apify/crawlee-python/pull/2159/files/f969b41bb44017d9e510ffd64e5d185f447f4602
https://github.com/janbuchar
janbuchar https://github.com/janbuchar
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.com/Mantisus
Mantisus https://github.com/Mantisus
ThrottlingRequestManager computes the wrong backoff for concurrent 429s and for Retry-After: 0 https://github.com/apify/crawlee-python/issues/2125
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2159
https://github.com/Mantisus
https://github.com/vdusek
https://github.com/janbuchar
https://github.com/apps/copilot-pull-request-reviewer
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.