René's URL Explorer Experiment


Title: feat: Add spider RequestDeduplicationMiddleware by jhg · Pull Request #261 · roach-php/core · GitHub

Open Graph Title: feat: Add spider RequestDeduplicationMiddleware by jhg · Pull Request #261 · roach-php/core

X Title: feat: Add spider RequestDeduplicationMiddleware by jhg · Pull Request #261 · roach-php/core

Description: The complete web scraping toolkit for PHP. Contribute to roach-php/core development by creating an account on GitHub.

Open Graph Description: I created it to avoid error about memory limit when it needs to crawl websites with many links. The downloader middleware late much to drop request, but this spider middleware drop it earlier then,...

X Description: I created it to avoid error about memory limit when it needs to crawl websites with many links. The downloader middleware late much to drop request, but this spider middleware drop it earlier then,...

Opengraph URL: https://github.com/roach-php/core/pull/261

X: @github

direct link

Domain: patch-diff.githubusercontent.com

route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:9de5fc1c-167b-0d4e-b7c8-8da242bf62b8
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id93DC:17F901:1042A30:14D0F8A:699190BD
html-safe-nonce39ef6b05c6aac5fe328b97d26cd89e3ac9de93f96c34abcc4b3328e7457c2e68
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5M0RDOjE3RjkwMToxMDQyQTMwOjE0RDBGOEE6Njk5MTkwQkQiLCJ2aXNpdG9yX2lkIjoiMTM5ODkwMDgxNjA2NTE3MTY0NSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac790cf90a0591785ac0919b58e648eab5e5786efaa18e92cba383950034198567
hovercard-subject-tagpull_request:2082805776
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/roach-php/core/261/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/6a05a898f2b2354725dfeab4b908b4ea380943f3f00c88029f2f09d84aad02a3/roach-php/core/pull/261
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/6a05a898f2b2354725dfeab4b908b4ea380943f3f00c88029f2f09d84aad02a3/roach-php/core/pull/261
og:image:altI created it to avoid error about memory limit when it needs to crawl websites with many links. The downloader middleware late much to drop request, but this spider middleware drop it earlier then,...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamejhg
hostnamegithub.com
expected-hostnamegithub.com
None42c603b9d642c4a9065a51770f75e5e27132fef0e858607f5c9cb7e422831a7b
turbo-cache-controlno-cache
go-importgithub.com/roach-php/core git https://github.com/roach-php/core.git
octolytics-dimension-user_id88873474
octolytics-dimension-user_loginroach-php
octolytics-dimension-repository_id397134075
octolytics-dimension-repository_nworoach-php/core
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id397134075
octolytics-dimension-repository_network_root_nworoach-php/core
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release848bc6032dcc93a9a7301dcc3f379a72ba13b96e
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://patch-diff.githubusercontent.com/roach-php/core/pull/261#start-of-content
https://patch-diff.githubusercontent.com/
Sign in https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2Froach-php%2Fcore%2Fpull%2F261
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2Froach-php%2Fcore%2Fpull%2F261
Sign up https://patch-diff.githubusercontent.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=roach-php%2Fcore
Reloadhttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
Reloadhttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
Reloadhttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
roach-php https://patch-diff.githubusercontent.com/roach-php
corehttps://patch-diff.githubusercontent.com/roach-php/core
Please reload this pagehttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
Notifications https://patch-diff.githubusercontent.com/login?return_to=%2Froach-php%2Fcore
Fork 80 https://patch-diff.githubusercontent.com/login?return_to=%2Froach-php%2Fcore
Star 1.5k https://patch-diff.githubusercontent.com/login?return_to=%2Froach-php%2Fcore
Code https://patch-diff.githubusercontent.com/roach-php/core
Issues 19 https://patch-diff.githubusercontent.com/roach-php/core/issues
Pull requests 14 https://patch-diff.githubusercontent.com/roach-php/core/pulls
Discussions https://patch-diff.githubusercontent.com/roach-php/core/discussions
Actions https://patch-diff.githubusercontent.com/roach-php/core/actions
Security 0 https://patch-diff.githubusercontent.com/roach-php/core/security
Insights https://patch-diff.githubusercontent.com/roach-php/core/pulse
Code https://patch-diff.githubusercontent.com/roach-php/core
Issues https://patch-diff.githubusercontent.com/roach-php/core/issues
Pull requests https://patch-diff.githubusercontent.com/roach-php/core/pulls
Discussions https://patch-diff.githubusercontent.com/roach-php/core/discussions
Actions https://patch-diff.githubusercontent.com/roach-php/core/actions
Security https://patch-diff.githubusercontent.com/roach-php/core/security
Insights https://patch-diff.githubusercontent.com/roach-php/core/pulse
jhghttps://patch-diff.githubusercontent.com/jhg
roach-php:mainhttps://patch-diff.githubusercontent.com/roach-php/core/tree/main
jhg:feat-spìder-request-deduplication-middlewarehttps://patch-diff.githubusercontent.com/jhg/roach-core/tree/feat-sp%C3%ACder-request-deduplication-middleware
Conversationhttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
Commits1 (1)https://patch-diff.githubusercontent.com/roach-php/core/pull/261/commits
Checkshttps://patch-diff.githubusercontent.com/roach-php/core/pull/261/checks
Files changedhttps://patch-diff.githubusercontent.com/roach-php/core/pull/261/files
feat: Add spider RequestDeduplicationMiddlewarehttps://patch-diff.githubusercontent.com/roach-php/core/pull/261#top
jhghttps://patch-diff.githubusercontent.com/jhg
roach-php:mainhttps://patch-diff.githubusercontent.com/roach-php/core/tree/main
jhg:feat-spìder-request-deduplication-middlewarehttps://patch-diff.githubusercontent.com/jhg/roach-core/tree/feat-sp%C3%ACder-request-deduplication-middleware
https://patch-diff.githubusercontent.com/jhg
jhghttps://patch-diff.githubusercontent.com/jhg
Sep 20, 2024https://patch-diff.githubusercontent.com/roach-php/core/pull/261#issue-2538384151
Please reload this pagehttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
https://patch-diff.githubusercontent.com/jhg
feat: Add spider RequestDeduplicationMiddlewarehttps://patch-diff.githubusercontent.com/roach-php/core/pull/261/commits/d31ab23adaed62a6f2f4450e9c5b4509e16c5de1
d31ab23https://patch-diff.githubusercontent.com/roach-php/core/pull/261/commits/d31ab23adaed62a6f2f4450e9c5b4509e16c5de1
https://patch-diff.githubusercontent.com/jhg
jhghttps://patch-diff.githubusercontent.com/jhg
ksassnowskihttps://patch-diff.githubusercontent.com/ksassnowski
code ownerhttps://patch-diff.githubusercontent.com/roach-php/core/blob/a4e6f159c7de8823a8483ee428cd258485795038/.github/CODEOWNERS#L1
September 20, 2024 09:52https://patch-diff.githubusercontent.com/roach-php/core/pull/261#event-14337452702
https://patch-diff.githubusercontent.com/jhg
jhghttps://patch-diff.githubusercontent.com/jhg
Sep 20, 2024https://patch-diff.githubusercontent.com/roach-php/core/pull/261#issuecomment-2363329259
Please reload this pagehttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
https://github.co/hiddenchars
https://patch-diff.githubusercontent.com/roach-php/core/pull/{{ revealButtonHref }}
Sign up for freehttps://patch-diff.githubusercontent.com/join?source=comment-repo
Sign in to commenthttps://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2Froach-php%2Fcore%2Fpull%2F261
https://patch-diff.githubusercontent.com/ksassnowski
ksassnowski https://patch-diff.githubusercontent.com/ksassnowski
Please reload this pagehttps://patch-diff.githubusercontent.com/roach-php/core/pull/261
https://patch-diff.githubusercontent.com/jhg
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.