René's URL Explorer Experiment


Title: HttpCrawler | API | Crawlee for Python · Fast, reliable Python web crawlers.

Open Graph Title: HttpCrawler | API | Crawlee for Python · Fast, reliable Python web crawlers.

Description: Specific version of generic `AbstractHttpCrawler`.

Open Graph Description: Specific version of generic `AbstractHttpCrawler`.

Opengraph URL: https://crawlee.dev/python/api/next/class/HttpCrawler

Generator: Docusaurus v3.10.2

direct link

Domain: crawlee.dev


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[]}

twitter:cardsummary_large_image
og:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
twitter:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
og:localeen
docusaurus_localeen
docsearch:languageen
docusaurus_versioncurrent
docusaurus_tagdocs-default-current
docsearch:versioncurrent
docsearch:docusaurus_tagdocs-default-current

Links:

Skip to main contenthttps://crawlee.dev/python/api/next/class/HttpCrawler#__docusaurus_skipToContent_fallback
Crawlee for Pythonhttps://crawlee.dev/python/
https://crawlee.dev/js
https://crawlee.dev/python/
https://crawlee.dev
Docshttps://crawlee.dev/python/docs/quick-start
APIhttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Bloghttps://crawlee.dev/blog
Nexthttps://crawlee.dev/python/api/next
Nexthttps://crawlee.dev/python/api/next
1.11https://crawlee.dev/python/api
0.6https://crawlee.dev/python/api/0.6
Get startedhttps://crawlee.dev/python/docs/quick-start
Crawlee for Pythonhttps://crawlee.dev/python/
Overviewhttps://crawlee.dev/python/api/next
Autoscalinghttps://crawlee.dev/python/api/next/class/AutoscaledPool
Browser managementhttps://crawlee.dev/python/api/next/class/BrowserController
Configurationhttps://crawlee.dev/python/api/next/class/ConcurrencySettings
Crawlershttps://crawlee.dev/python/api/next/class/AbstractHttpCrawler
AbstractHttpCrawlerhttps://crawlee.dev/python/api/next/class/AbstractHttpCrawler
AdaptivePlaywrightCrawlerhttps://crawlee.dev/python/api/next/class/AdaptivePlaywrightCrawler
BasicCrawlerhttps://crawlee.dev/python/api/next/class/BasicCrawler
BeautifulSoupCrawlerhttps://crawlee.dev/python/api/next/class/BeautifulSoupCrawler
FileDownloadCrawlerhttps://crawlee.dev/python/api/next/class/FileDownloadCrawler
HttpCrawlerhttps://crawlee.dev/python/api/next/class/HttpCrawler
ParselCrawlerhttps://crawlee.dev/python/api/next/class/ParselCrawler
PlaywrightCrawlerhttps://crawlee.dev/python/api/next/class/PlaywrightCrawler
PydanticAiCrawlerhttps://crawlee.dev/python/api/next/class/PydanticAiCrawler
StagehandCrawlerhttps://crawlee.dev/python/api/next/class/StagehandCrawler
Crawling contextshttps://crawlee.dev/python/api/next/class/AdaptivePlaywrightCrawlingContext
Errorshttps://crawlee.dev/python/api/next/class/ContextPipelineFinalizationError
Event datahttps://crawlee.dev/python/api/next/enum/Event
Event managershttps://crawlee.dev/python/api/next/class/EventManager
Functionshttps://crawlee.dev/python/api/next/class/AddRequestsFunction
HTTP clientshttps://crawlee.dev/python/api/next/class/CurlImpersonateHttpClient
HTTP parsershttps://crawlee.dev/python/api/next/class/AbstractHttpParser
Request loadershttps://crawlee.dev/python/api/next/class/RequestList
Session managementhttps://crawlee.dev/python/api/next/class/CookieParam
Statisticshttps://crawlee.dev/python/api/next/class/AdaptivePlaywrightCrawlerStatisticState
Storage clientshttps://crawlee.dev/python/api/next/class/FileSystemStorageClient
Storage datahttps://crawlee.dev/python/api/next/class/AddRequestsResponse
Storageshttps://crawlee.dev/python/api/next/class/Dataset
Otherhttps://crawlee.dev/python/api/next/class/BasePydanticAiHtmlDistiller
1.11https://crawlee.dev/python/api
https://crawlee.dev/python/
AbstractHttpCrawlerhttps://crawlee.dev/python/api/next/class/AbstractHttpCrawler
https://crawlee.dev/python/api/next/class/HttpCrawler#Index
__init__https://crawlee.dev/python/api/next/class/HttpCrawler#__init__
add_requestshttps://crawlee.dev/python/api/next/class/HttpCrawler#add_requests
create_parsed_http_crawler_classhttps://crawlee.dev/python/api/next/class/HttpCrawler#create_parsed_http_crawler_class
error_handlerhttps://crawlee.dev/python/api/next/class/HttpCrawler#error_handler
export_datahttps://crawlee.dev/python/api/next/class/HttpCrawler#export_data
failed_request_handlerhttps://crawlee.dev/python/api/next/class/HttpCrawler#failed_request_handler
get_datahttps://crawlee.dev/python/api/next/class/HttpCrawler#get_data
get_datasethttps://crawlee.dev/python/api/next/class/HttpCrawler#get_dataset
get_key_value_storehttps://crawlee.dev/python/api/next/class/HttpCrawler#get_key_value_store
get_request_managerhttps://crawlee.dev/python/api/next/class/HttpCrawler#get_request_manager
on_skipped_requesthttps://crawlee.dev/python/api/next/class/HttpCrawler#on_skipped_request
post_navigation_hookhttps://crawlee.dev/python/api/next/class/HttpCrawler#post_navigation_hook
pre_navigation_hookhttps://crawlee.dev/python/api/next/class/HttpCrawler#pre_navigation_hook
runhttps://crawlee.dev/python/api/next/class/HttpCrawler#run
stophttps://crawlee.dev/python/api/next/class/HttpCrawler#stop
use_statehttps://crawlee.dev/python/api/next/class/HttpCrawler#use_state
loghttps://crawlee.dev/python/api/next/class/HttpCrawler#log
routerhttps://crawlee.dev/python/api/next/class/HttpCrawler#router
statisticshttps://crawlee.dev/python/api/next/class/HttpCrawler#statistics
https://crawlee.dev/python/api/next/class/HttpCrawler#Methods
https://crawlee.dev/python/api/next/class/HttpCrawler#__init__
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_http/_http_crawler.py#L49
AbstractHttpCrawler.__init__https://crawlee.dev/python/api/next/class/AbstractHttpCrawler#__init__
https://crawlee.dev/python/api/next/class/HttpCrawler#add_requests
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L855
BasicCrawler.add_requestshttps://crawlee.dev/python/api/next/class/BasicCrawler#add_requests
https://crawlee.dev/python/api/next/class/HttpCrawler#create_parsed_http_crawler_class
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py#L93
AbstractHttpCrawler.create_parsed_http_crawler_classhttps://crawlee.dev/python/api/next/class/AbstractHttpCrawler#create_parsed_http_crawler_class
https://crawlee.dev/python/api/next/class/HttpCrawler#error_handler
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L665
BasicCrawler.error_handlerhttps://crawlee.dev/python/api/next/class/BasicCrawler#error_handler
https://crawlee.dev/python/api/next/class/HttpCrawler#export_data
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L944
BasicCrawler.export_datahttps://crawlee.dev/python/api/next/class/BasicCrawler#export_data
https://crawlee.dev/python/api/next/class/HttpCrawler#failed_request_handler
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L675
BasicCrawler.failed_request_handlerhttps://crawlee.dev/python/api/next/class/BasicCrawler#failed_request_handler
https://crawlee.dev/python/api/next/class/HttpCrawler#get_data
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L914
DatasetItemsListPagehttps://crawlee.dev/python/api/next/class/DatasetItemsListPage
BasicCrawler.get_datahttps://crawlee.dev/python/api/next/class/BasicCrawler#get_data
DatasetItemsListPagehttps://crawlee.dev/python/api/next/class/DatasetItemsListPage
https://crawlee.dev/python/api/next/class/HttpCrawler#get_dataset
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L633
Datasethttps://crawlee.dev/python/api/next/class/Dataset
BasicCrawler.get_datasethttps://crawlee.dev/python/api/next/class/BasicCrawler#get_dataset
Datasethttps://crawlee.dev/python/api/next/class/Dataset
https://crawlee.dev/python/api/next/class/HttpCrawler#get_key_value_store
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L649
KeyValueStorehttps://crawlee.dev/python/api/next/class/KeyValueStore
BasicCrawler.get_key_value_storehttps://crawlee.dev/python/api/next/class/BasicCrawler#get_key_value_store
KeyValueStorehttps://crawlee.dev/python/api/next/class/KeyValueStore
https://crawlee.dev/python/api/next/class/HttpCrawler#get_request_manager
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L623
RequestManagerhttps://crawlee.dev/python/api/next/class/RequestManager
BasicCrawler.get_request_managerhttps://crawlee.dev/python/api/next/class/BasicCrawler#get_request_manager
RequestManagerhttps://crawlee.dev/python/api/next/class/RequestManager
https://crawlee.dev/python/api/next/class/HttpCrawler#on_skipped_request
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L685
SkippedRequestCallbackhttps://crawlee.dev/python/api/next#SkippedRequestCallback
BasicCrawler.on_skipped_requesthttps://crawlee.dev/python/api/next/class/BasicCrawler#on_skipped_request
SkippedRequestCallbackhttps://crawlee.dev/python/api/next#SkippedRequestCallback
SkippedRequestCallbackhttps://crawlee.dev/python/api/next#SkippedRequestCallback
https://crawlee.dev/python/api/next/class/HttpCrawler#post_navigation_hook
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py#L350
AbstractHttpCrawler.post_navigation_hookhttps://crawlee.dev/python/api/next/class/AbstractHttpCrawler#post_navigation_hook
https://crawlee.dev/python/api/next/class/HttpCrawler#pre_navigation_hook
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py#L342
AbstractHttpCrawler.pre_navigation_hookhttps://crawlee.dev/python/api/next/class/AbstractHttpCrawler#pre_navigation_hook
https://crawlee.dev/python/api/next/class/HttpCrawler#run
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L693
FinalStatisticshttps://crawlee.dev/python/api/next/class/FinalStatistics
BasicCrawler.runhttps://crawlee.dev/python/api/next/class/BasicCrawler#run
FinalStatisticshttps://crawlee.dev/python/api/next/class/FinalStatistics
https://crawlee.dev/python/api/next/class/HttpCrawler#stop
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L545
BasicCrawler.stophttps://crawlee.dev/python/api/next/class/BasicCrawler#stop
https://crawlee.dev/python/api/next/class/HttpCrawler#use_state
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L903
BasicCrawler.use_statehttps://crawlee.dev/python/api/next/class/BasicCrawler#use_state
https://crawlee.dev/python/api/next/class/HttpCrawler#Properties
https://crawlee.dev/python/api/next/class/HttpCrawler#log
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L521
BasicCrawler.loghttps://crawlee.dev/python/api/next/class/BasicCrawler#log
https://crawlee.dev/python/api/next/class/HttpCrawler#router
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L526
BasicCrawler.routerhttps://crawlee.dev/python/api/next/class/BasicCrawler#router
BasicCrawler.routerhttps://crawlee.dev/python/api/next/class/BasicCrawler#router
https://crawlee.dev/python/api/next/class/HttpCrawler#statistics
https://github.com/apify/crawlee-python/blob/7bd32e7d9cfe219f3aebc13cf631b45fdb14e653//src/crawlee/crawlers/_basic/_basic_crawler.py#L541
BasicCrawler.statisticshttps://crawlee.dev/python/api/next/class/BasicCrawler#statistics
PreviousFileDownloadCrawlerhttps://crawlee.dev/python/api/next/class/FileDownloadCrawler
NextParselCrawlerhttps://crawlee.dev/python/api/next/class/ParselCrawler
__init__https://crawlee.dev/python/api/next/class/HttpCrawler#__init__
add_requestshttps://crawlee.dev/python/api/next/class/HttpCrawler#add_requests
create_parsed_http_crawler_classhttps://crawlee.dev/python/api/next/class/HttpCrawler#create_parsed_http_crawler_class
error_handlerhttps://crawlee.dev/python/api/next/class/HttpCrawler#error_handler
export_datahttps://crawlee.dev/python/api/next/class/HttpCrawler#export_data
failed_request_handlerhttps://crawlee.dev/python/api/next/class/HttpCrawler#failed_request_handler
get_datahttps://crawlee.dev/python/api/next/class/HttpCrawler#get_data
get_datasethttps://crawlee.dev/python/api/next/class/HttpCrawler#get_dataset
get_key_value_storehttps://crawlee.dev/python/api/next/class/HttpCrawler#get_key_value_store
get_request_managerhttps://crawlee.dev/python/api/next/class/HttpCrawler#get_request_manager
on_skipped_requesthttps://crawlee.dev/python/api/next/class/HttpCrawler#on_skipped_request
post_navigation_hookhttps://crawlee.dev/python/api/next/class/HttpCrawler#post_navigation_hook
pre_navigation_hookhttps://crawlee.dev/python/api/next/class/HttpCrawler#pre_navigation_hook
runhttps://crawlee.dev/python/api/next/class/HttpCrawler#run
stophttps://crawlee.dev/python/api/next/class/HttpCrawler#stop
use_statehttps://crawlee.dev/python/api/next/class/HttpCrawler#use_state
loghttps://crawlee.dev/python/api/next/class/HttpCrawler#log
routerhttps://crawlee.dev/python/api/next/class/HttpCrawler#router
statisticshttps://crawlee.dev/python/api/next/class/HttpCrawler#statistics
https://crawlee.dev
Quick starthttps://crawlee.dev/python/docs/quick-start
Guideshttps://crawlee.dev/python/docs/guides
API referencehttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Discordhttps://discord.com/invite/jyEM2PRvMU
Stack Overflowhttps://stackoverflow.com/questions/tagged/crawlee-python
Twitterhttps://twitter.com/apify
YouTubehttps://www.youtube.com/apify
Apify platformhttps://apify.com
Docusaurushttps://docusaurus.io
GitHubhttps://github.com/apify/crawlee-python

Viewport: width=device-width, initial-scale=1.0


URLs of crawlers that visited me.