René's URL Explorer Experiment


Title: GitHub - CloakHQ/crawlee-python: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation. · GitHub

Open Graph Title: GitHub - CloakHQ/crawlee-python: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

X Title: GitHub - CloakHQ/crawlee-python: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation. - CloakHQ/crawlee-python

Open Graph Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Wo...

X Description: Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Wo...

Opengraph URL: https://github.com/CloakHQ/crawlee-python

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository
route-controllerfiles
route-actiondisambiguate
fetch-noncev2:00b772ba-d03e-da04-94d5-29d33f2c6e3d
current-catalog-service-hashf3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id83DE:3C5364:222CC1:2C62D4:6AEBEDD7
html-safe-nonced27fe9a76acb52263a6466fd1654a072a3949a689531a58be3327649948b5730
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4M0RFOjNDNTM2NDoyMjJDQzE6MkM2MkQ0OjZBRUJFREQ3IiwidmlzaXRvcl9pZCI6Ijc4MjM2NTE3OTE4MzkxNjE4MTUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac90df655f6cc5f560888229bd7f12028dcc2c780cf2e066321f0d6d44b9458a1d
hovercard-subject-tagrepository:1182129517
github-keyboard-shortcutsrepository,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location//
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/CloakHQ/crawlee-python
twitter:imagehttps://opengraph.githubassets.com/546e64db327adb78b5ba2ca5186de36f2085b21595cac69032dbe2ef35d1e987/CloakHQ/crawlee-python
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/546e64db327adb78b5ba2ca5186de36f2085b21595cac69032dbe2ef35d1e987/CloakHQ/crawlee-python
og:image:altCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Wo...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None8ec2990e713e4b47a35fdedb81d463ccadec4d3a4ba7253c2513727b0a58ecf9
turbo-cache-controlno-cache
go-importgithub.com/CloakHQ/crawlee-python git https://github.com/CloakHQ/crawlee-python.git
octolytics-dimension-user_id263128610
octolytics-dimension-user_loginCloakHQ
octolytics-dimension-repository_id1182129517
octolytics-dimension-repository_nwoCloakHQ/crawlee-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forktrue
octolytics-dimension-repository_parent_id741365632
octolytics-dimension-repository_parent_nwoapify/crawlee-python
octolytics-dimension-repository_network_root_id741365632
octolytics-dimension-repository_network_root_nwoapify/crawlee-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releaseddbc1c308788e31a0779acbd2fdbf652c948e2ef
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/CloakHQ/crawlee-python#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FCloakHQ%2Fcrawlee-python
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FCloakHQ%2Fcrawlee-python
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=CloakHQ%2Fcrawlee-python
Reloadhttps://github.com/CloakHQ/crawlee-python
Reloadhttps://github.com/CloakHQ/crawlee-python
Reloadhttps://github.com/CloakHQ/crawlee-python
CloakHQhttps://github.com/CloakHQ
crawlee-pythonhttps://github.com/CloakHQ/crawlee-python
Codehttps://github.com/CloakHQ/crawlee-python
Pull requestshttps://github.com/CloakHQ/crawlee-python/pulls
Actionshttps://github.com/CloakHQ/crawlee-python/actions
Projectshttps://github.com/CloakHQ/crawlee-python/projects
Security and qualityhttps://github.com/CloakHQ/crawlee-python/security
Insightshttps://github.com/CloakHQ/crawlee-python/pulse
https://github.com/CloakHQ/crawlee-python
crawlee-pythonhttps://github.com/CloakHQ/crawlee-python
apify/crawlee-pythonhttps://github.com/apify/crawlee-python
Notificationshttps://github.com/login?return_to=%2FCloakHQ%2Fcrawlee-python
Fork1 (1)https://github.com/login?return_to=%2FCloakHQ%2Fcrawlee-python
Star7 (7)https://github.com/login?return_to=%2FCloakHQ%2Fcrawlee-python
crawlee.dev/python/https://crawlee.dev/python/
Readmehttps://github.com/CloakHQ/crawlee-python#readme-ov-file
Licensehttps://github.com/CloakHQ/crawlee-python#License-1-ov-file
Contributinghttps://github.com/CloakHQ/crawlee-python#contributing-ov-file
Activityhttps://github.com/CloakHQ/crawlee-python/activity
Custom propertieshttps://github.com/CloakHQ/crawlee-python/custom-properties
1 forkhttps://github.com/CloakHQ/crawlee-python/forks
Report repositoryhttps://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FCloakHQ%2Fcrawlee-python&report=CloakHQ+%28user%29
Brancheshttps://github.com/CloakHQ/crawlee-python/branches
Tagshttps://github.com/CloakHQ/crawlee-python/tags
https://github.com/CloakHQ/crawlee-python/branches
https://github.com/CloakHQ/crawlee-python/tags
1,407 Commitshttps://github.com/CloakHQ/crawlee-python/commits/master/
https://github.com/CloakHQ/crawlee-python/commits/master/
.githubhttps://github.com/CloakHQ/crawlee-python/tree/master/.github
.githubhttps://github.com/CloakHQ/crawlee-python/tree/master/.github
docshttps://github.com/CloakHQ/crawlee-python/tree/master/docs
docshttps://github.com/CloakHQ/crawlee-python/tree/master/docs
src/crawleehttps://github.com/CloakHQ/crawlee-python/tree/master/src/crawlee
src/crawleehttps://github.com/CloakHQ/crawlee-python/tree/master/src/crawlee
testshttps://github.com/CloakHQ/crawlee-python/tree/master/tests
testshttps://github.com/CloakHQ/crawlee-python/tree/master/tests
websitehttps://github.com/CloakHQ/crawlee-python/tree/master/website
websitehttps://github.com/CloakHQ/crawlee-python/tree/master/website
.editorconfighttps://github.com/CloakHQ/crawlee-python/blob/master/.editorconfig
.editorconfighttps://github.com/CloakHQ/crawlee-python/blob/master/.editorconfig
.gitignorehttps://github.com/CloakHQ/crawlee-python/blob/master/.gitignore
.gitignorehttps://github.com/CloakHQ/crawlee-python/blob/master/.gitignore
.markdownlint.yamlhttps://github.com/CloakHQ/crawlee-python/blob/master/.markdownlint.yaml
.markdownlint.yamlhttps://github.com/CloakHQ/crawlee-python/blob/master/.markdownlint.yaml
.pre-commit-config.yamlhttps://github.com/CloakHQ/crawlee-python/blob/master/.pre-commit-config.yaml
.pre-commit-config.yamlhttps://github.com/CloakHQ/crawlee-python/blob/master/.pre-commit-config.yaml
.rules.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/.rules.md
.rules.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/.rules.md
AGENTS.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/AGENTS.md
AGENTS.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/AGENTS.md
CHANGELOG.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/CHANGELOG.md
CHANGELOG.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/CHANGELOG.md
CLAUDE.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/CLAUDE.md
CLAUDE.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/CLAUDE.md
CONTRIBUTING.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/CONTRIBUTING.md
CONTRIBUTING.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/CONTRIBUTING.md
GEMINI.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/GEMINI.md
GEMINI.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/GEMINI.md
LICENSEhttps://github.com/CloakHQ/crawlee-python/blob/master/LICENSE
LICENSEhttps://github.com/CloakHQ/crawlee-python/blob/master/LICENSE
README.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/README.md
README.mdhttps://github.com/CloakHQ/crawlee-python/blob/master/README.md
codecov.yamlhttps://github.com/CloakHQ/crawlee-python/blob/master/codecov.yaml
codecov.yamlhttps://github.com/CloakHQ/crawlee-python/blob/master/codecov.yaml
pyproject.tomlhttps://github.com/CloakHQ/crawlee-python/blob/master/pyproject.toml
pyproject.tomlhttps://github.com/CloakHQ/crawlee-python/blob/master/pyproject.toml
renovate.jsonhttps://github.com/CloakHQ/crawlee-python/blob/master/renovate.json
renovate.jsonhttps://github.com/CloakHQ/crawlee-python/blob/master/renovate.json
typos.tomlhttps://github.com/CloakHQ/crawlee-python/blob/master/typos.toml
typos.tomlhttps://github.com/CloakHQ/crawlee-python/blob/master/typos.toml
uv.lockhttps://github.com/CloakHQ/crawlee-python/blob/master/uv.lock
uv.lockhttps://github.com/CloakHQ/crawlee-python/blob/master/uv.lock
READMEhttps://github.com/CloakHQ/crawlee-python
Contributinghttps://github.com/CloakHQ/crawlee-python
Licensehttps://github.com/CloakHQ/crawlee-python
https://crawlee.dev
https://github.com/CloakHQ/crawlee-python#----------------------------------------------------a-web-scraping-and-browser-automation-library
https://trendshift.io/repositories/11169
https://badge.fury.io/py/crawlee
https://pypi.org/project/crawlee/
https://codecov.io/gh/apify/crawlee-python
https://pypi.org/project/crawlee/
https://discord.gg/jyEM2PRvMU
Crawlee project websitehttps://crawlee.dev/python/
Crawlee for JS/TS on GitHubhttps://github.com/apify/crawlee
https://github.com/CloakHQ/crawlee-python#installation
Introduction tutorialhttps://crawlee.dev/python/docs/introduction
crawleehttps://pypi.org/project/crawlee/
Playwrighthttps://playwright.dev/
Setting uphttps://crawlee.dev/python/docs/introduction/setting-up
https://github.com/CloakHQ/crawlee-python#with-crawlee-cli
uvhttps://pypi.org/project/uv/
uvhttps://pypi.org/project/uv/
installation guidehttps://docs.astral.sh/uv/getting-started/installation/
https://github.com/CloakHQ/crawlee-python#examples
https://github.com/CloakHQ/crawlee-python#beautifulsoupcrawler
BeautifulSoupCrawlerhttps://crawlee.dev/python/api/class/BeautifulSoupCrawler
HttpxHttpClienthttps://crawlee.dev/python/api/class/HttpxHttpClient
BeautifulSouphttps://pypi.org/project/beautifulsoup4/
PlaywrightCrawlerhttps://crawlee.dev/python/api/class/PlaywrightCrawler
https://github.com/CloakHQ/crawlee-python#playwrightcrawler
PlaywrightCrawlerhttps://crawlee.dev/python/api/class/PlaywrightCrawler
Playwrighthttps://playwright.dev/
BeautifulSoupCrawlerhttps://crawlee.dev/python/api/class/BeautifulSoupCrawler
https://github.com/CloakHQ/crawlee-python#more-examples
Exampleshttps://crawlee.dev/python/docs/examples
https://github.com/CloakHQ/crawlee-python#features
https://github.com/CloakHQ/crawlee-python#why-use-crawlee-instead-of-just-a-random-http-library-with-an-html-parser
https://github.com/CloakHQ/crawlee-python#why-to-use-crawlee-rather-than-scrapy
Asynciohttps://docs.python.org/3/library/asyncio.html
datasetshttps://crawlee.dev/python/api/class/Dataset
key-value storeshttps://crawlee.dev/python/api/class/KeyValueStore
https://github.com/CloakHQ/crawlee-python#running-on-the-apify-platform
Apifyhttps://apify.com
Apify SDK websitehttps://docs.apify.com/sdk/python/
https://github.com/CloakHQ/crawlee-python#support
submit an issue on GitHubhttps://github.com/apify/crawlee-python/issues
Stack Overflowhttps://stackoverflow.com/questions/tagged/apify
Discord serverhttps://discord.com/invite/jyEM2PRvMU
https://github.com/CloakHQ/crawlee-python#contributing
CONTRIBUTING.mdhttps://github.com/apify/crawlee-python/blob/master/CONTRIBUTING.md
https://github.com/CloakHQ/crawlee-python#license
LICENSEhttps://github.com/apify/crawlee-python/blob/master/LICENSE
crawlee.dev/python/https://crawlee.dev/python/
Readmehttps://github.com/CloakHQ/crawlee-python#readme-ov-file
Licensehttps://github.com/CloakHQ/crawlee-python#License-1-ov-file
Contributinghttps://github.com/CloakHQ/crawlee-python#contributing-ov-file
Activityhttps://github.com/CloakHQ/crawlee-python/activity
Custom propertieshttps://github.com/CloakHQ/crawlee-python/custom-properties
1 forkhttps://github.com/CloakHQ/crawlee-python/forks
Report repositoryhttps://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FCloakHQ%2Fcrawlee-python&report=CloakHQ+%28user%29
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.