René's URL Explorer Experiment


Title: fix(crawlers): Keep the crawler usable after a failed run by vdusek · Pull Request #2255 · apify/crawlee-python · GitHub

Open Graph Title: fix(crawlers): Keep the crawler usable after a failed run by vdusek · Pull Request #2255 · apify/crawlee-python

X Title: fix(crawlers): Keep the crawler usable after a failed run by vdusek · Pull Request #2255 · apify/crawlee-python

Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included. - fix(crawlers): Keep the crawler usable after a failed run by vdusek · Pull Request #2255 · apify/crawlee-python

Open Graph Description: Follow-up to #2229, split out of #2239. Crawler state after a failed run An exception during setup (e.g. add_requests) left _running set, so every later run() raised RuntimeError. The whole run bo...

X Description: Follow-up to #2229, split out of #2239. Crawler state after a failed run An exception during setup (e.g. add_requests) left _running set, so every later run() raised RuntimeError. The whole run bo...

Opengraph URL: https://github.com/apify/crawlee-python/pull/2255

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:b62dbe23-4bcf-7486-0099-0f6124103262
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idA91E:395430:ABE931:E4CC25:6B00EB45
html-safe-nonce625f0e4a2e7a87d52a3246be03428ec083782c98db9e3c9357aee1c9720d7627
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBOTFFOjM5NTQzMDpBQkU5MzE6RTRDQzI1OjZCMDBFQjQ1IiwidmlzaXRvcl9pZCI6IjIwMTU3NDIwMzA1NzA5MDg0ODUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac23a810ce9034b290e8512fd7141e97a006ed4079545f9dda6efac3fe2beb8fe3
hovercard-subject-tagpull_request:4625607236
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/apify/crawlee-python/2255/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/12ab5a30e979ccff4e5f81a1fba6783fa866008f809950f5c3625884e637fe1e/apify/crawlee-python/pull/2255
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/12ab5a30e979ccff4e5f81a1fba6783fa866008f809950f5c3625884e637fe1e/apify/crawlee-python/pull/2255
og:image:altFollow-up to #2229, split out of #2239. Crawler state after a failed run An exception during setup (e.g. add_requests) left _running set, so every later run() raised RuntimeError. The whole run bo...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamevdusek
hostnamegithub.com
expected-hostnamegithub.com
None4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc
turbo-cache-controlno-cache
go-importgithub.com/apify/crawlee-python git https://github.com/apify/crawlee-python.git
octolytics-dimension-user_id24586296
octolytics-dimension-user_loginapify
octolytics-dimension-repository_id741365632
octolytics-dimension-repository_nwoapify/crawlee-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id741365632
octolytics-dimension-repository_network_root_nwoapify/crawlee-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release9983dd5da06943f833c971dd5b6bc988e2858be5
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-namepull-requests

Links:

Skip to contenthttps://github.com/apify/crawlee-python/pull/2255#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2255
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2255
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=apify%2Fcrawlee-python
Reloadhttps://github.com/apify/crawlee-python/pull/2255
Reloadhttps://github.com/apify/crawlee-python/pull/2255
Reloadhttps://github.com/apify/crawlee-python/pull/2255
apifyhttps://github.com/apify
crawlee-pythonhttps://github.com/apify/crawlee-python
Codehttps://github.com/apify/crawlee-python
Issues72 (72)https://github.com/apify/crawlee-python/issues
Pull requests14 (14)https://github.com/apify/crawlee-python/pulls
Discussionshttps://github.com/apify/crawlee-python/discussions
Actionshttps://github.com/apify/crawlee-python/actions
Security and quality1 (1)https://github.com/apify/crawlee-python/security
Insightshttps://github.com/apify/crawlee-python/insights
https://github.com/apify/crawlee-python
vdusekhttps://github.com/vdusek
masterhttps://github.com/apify/crawlee-python/tree/master
fix/crawler-failed-runhttps://github.com/apify/crawlee-python/tree/fix/crawler-failed-run
Conversationhttps://github.com/apify/crawlee-python/pull/2255
Commits3 (3)https://github.com/apify/crawlee-python/pull/2255/commits
Checkshttps://github.com/apify/crawlee-python/pull/2255/checks
Files changedhttps://github.com/apify/crawlee-python/pull/2255/files
fix(crawlers): Keep the crawler usable after a failed runhttps://github.com/apify/crawlee-python/pull/2255#top
vdusekhttps://github.com/vdusek
masterhttps://github.com/apify/crawlee-python/tree/master
fix/crawler-failed-runhttps://github.com/apify/crawlee-python/tree/fix/crawler-failed-run
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#issue-5568024770
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2255
#2229https://github.com/apify/crawlee-python/pull/2229
#2239https://github.com/apify/crawlee-python/pull/2239
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2255
vdusekhttps://github.com/vdusek
September 24, 2026 14:04https://github.com/apify/crawlee-python/pull/2255#commits-pushed-accd893
https://github.com/vdusek
fix(crawlers): Keep the crawler usable and its pending requests after…https://github.com/apify/crawlee-python/pull/2255/commits/accd89370d643a3c378fbcb9265757f39f8458e5
accd893https://github.com/apify/crawlee-python/pull/2255/commits/accd89370d643a3c378fbcb9265757f39f8458e5
https://github.com/vdusek
fix(crawlers): Mark a run failed when its post-crawl steps raisehttps://github.com/apify/crawlee-python/pull/2255/commits/7bf7a83a1e6e3510b5af7a5fc17638e1f12c207b
7bf7a83https://github.com/apify/crawlee-python/pull/2255/commits/7bf7a83a1e6e3510b5af7a5fc17638e1f12c207b
https://github.com/vdusek
test(autoscaling): Cover the orchestrator error logged on a cancelled…https://github.com/apify/crawlee-python/pull/2255/commits/f3c83e5493ba8b87d23c9d39d96a1a839b8bd66e
f3c83e5https://github.com/apify/crawlee-python/pull/2255/commits/f3c83e5493ba8b87d23c9d39d96a1a839b8bd66e
https://github.com/vdusek
vdusekhttps://github.com/vdusek
t-tooling https://github.com/apify/crawlee-python/issues?q=state%3Aopen%20label%3At-tooling
adhoc https://github.com/apify/crawlee-python/issues?q=state%3Aopen%20label%3Aadhoc
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#event-31753888409
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#event-31753890651
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 24, 2026 https://github.com/apify/crawlee-python/pull/2255#ref-pullrequest-5474398499
fix(browsers): Retry the temp directory removal until the browser releases its files #2239 https://github.com/apify/crawlee-python/pull/2239
https://github.com/apps/codecov
codecovhttps://github.com/apps/codecov
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#issuecomment-5813795258
Codecovhttps://app.codecov.io/gh/apify/crawlee-python/pull/2255?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
45bb0e2https://app.codecov.io/gh/apify/crawlee-python/commit/45bb0e25df8fdad3caf0e756e32c2063717da4ce?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
f3c83e5https://app.codecov.io/gh/apify/crawlee-python/commit/f3c83e5493ba8b87d23c9d39d96a1a839b8bd66e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
Files with missing lineshttps://app.codecov.io/gh/apify/crawlee-python/pull/2255?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
src/crawlee/crawlers/_basic/_basic_crawler.pyhttps://app.codecov.io/gh/apify/crawlee-python/pull/2255?src=pr&el=tree&filepath=src%2Fcrawlee%2Fcrawlers%2F_basic%2F_basic_crawler.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify#diff-c3JjL2NyYXdsZWUvY3Jhd2xlcnMvX2Jhc2ljL19iYXNpY19jcmF3bGVyLnB5
6 Missing ⚠️ https://app.codecov.io/gh/apify/crawlee-python/pull/2255?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
Flaghttps://app.codecov.io/gh/apify/crawlee-python/pull/2255/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
unithttps://app.codecov.io/gh/apify/crawlee-python/pull/2255/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
Click herehttps://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify#carryforward-flags-in-the-pull-request-comment
☔ View full report in Codecov by Harnesshttps://app.codecov.io/gh/apify/crawlee-python/pull/2255?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
Share it herehttps://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apify
Test Analyticshttps://docs.codecov.com/docs/test-analytics
JS Bundle Analysishttps://docs.codecov.com/docs/javascript-bundle-analysis
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2255
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#event-31757376334
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#event-31757448584
https://github.com/vdusek
vdusekhttps://github.com/vdusek
Pijukatelhttps://github.com/Pijukatel
September 24, 2026 13:06https://github.com/apify/crawlee-python/pull/2255#event-31758150514
https://github.com/vdusek
vdusekhttps://github.com/vdusek
September 24, 2026 13:06https://github.com/apify/crawlee-python/pull/2255#event-31758154410
https://github.com/apify-service-account
apify-service-accounthttps://github.com/apify-service-account
tested https://github.com/apify/crawlee-python/issues?q=state%3Aopen%20label%3Atested
Sep 24, 2026https://github.com/apify/crawlee-python/pull/2255#event-31758179211
https://github.com/Pijukatel
Pijukatelhttps://github.com/Pijukatel
Sep 29, 2026 https://github.com/apify/crawlee-python/pull/2255#pullrequestreview-5348799898
View reviewed changes https://github.com/apify/crawlee-python/pull/2255/files/f3c83e5493ba8b87d23c9d39d96a1a839b8bd66e
https://github.com/vdusek
vdusekhttps://github.com/vdusek
3e73166https://github.com/apify/crawlee-python/commit/3e73166fa00160c4c1df656e9a25f1c60ae89773
Sep 29, 2026https://github.com/apify/crawlee-python/pull/2255#event-32058206851
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2255
https://github.com/vdusek
vdusekhttps://github.com/vdusek
September 29, 2026 07:19https://github.com/apify/crawlee-python/pull/2255#event-32058209639
vdusekhttps://github.com/vdusek
Sep 29, 2026 https://github.com/apify/crawlee-python/pull/2255#ref-commit-8f41522
https://github.com/vdusek
fix(browsers): Retry the temp directory removal until the browser rel…https://github.com/apify/crawlee-python/commit/8f41522fa9a15bfd585ef9745c853c25c26d9074
8f41522https://github.com/apify/crawlee-python/commit/8f41522fa9a15bfd585ef9745c853c25c26d9074
#2239https://github.com/apify/crawlee-python/pull/2239
https://github.com/apify/crawlee-python/actions/runs/35085779018https://github.com/apify/crawlee-python/actions/runs/35085779018
https://github.com/apify/crawlee-python/actions/runs/35991074349/job/107605022234https://github.com/apify/crawlee-python/actions/runs/35991074349/job/107605022234
#2255https://github.com/apify/crawlee-python/pull/2255
https://github.co/hiddenchars
https://github.com/apify/crawlee-python/pull/{{ revealButtonHref }}
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapify%2Fcrawlee-python%2Fpull%2F2255
https://github.com/Pijukatel
Pijukatel https://github.com/Pijukatel
https://github.com/apify/crawlee-python/pull/2255/files/f3c83e5493ba8b87d23c9d39d96a1a839b8bd66e
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2255
https://github.com/vdusek
vdusek https://github.com/vdusek
adhoc https://github.com/apify/crawlee-python/issues?q=state%3Aopen%20label%3Aadhoc
t-tooling https://github.com/apify/crawlee-python/issues?q=state%3Aopen%20label%3At-tooling
tested https://github.com/apify/crawlee-python/issues?q=state%3Aopen%20label%3Atested
Please reload this pagehttps://github.com/apify/crawlee-python/pull/2255
https://github.com/vdusek
https://github.com/Pijukatel
https://github.com/apify-service-account
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.