René's URL Explorer Experiment


Title: HttpCrawlingContext | API | Crawlee for Python · Fast, reliable Python web crawlers.

Open Graph Title: HttpCrawlingContext | API | Crawlee for Python · Fast, reliable Python web crawlers.

Description: The crawling context used by the `AbstractHttpCrawler`.

Open Graph Description: The crawling context used by the `AbstractHttpCrawler`.

Opengraph URL: https://crawlee.dev/python/api/class/HttpCrawlingContext

Generator: Docusaurus v3.10.2

direct link

Domain: crawlee.dev


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[]}

twitter:cardsummary_large_image
og:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
twitter:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
og:localeen
docusaurus_localeen
docsearch:languageen
docusaurus_version1.11
docusaurus_tagdocs-default-1.11
docsearch:version1.11
docsearch:docusaurus_tagdocs-default-1.11

Links:

Skip to main contenthttps://crawlee.dev/python/api/class/HttpCrawlingContext#__docusaurus_skipToContent_fallback
Crawlee for Pythonhttps://crawlee.dev/python/
https://crawlee.dev/js
https://crawlee.dev/python/
https://crawlee.dev
Docshttps://crawlee.dev/python/docs/quick-start
APIhttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Bloghttps://crawlee.dev/blog
1.11https://crawlee.dev/python/api
Nexthttps://crawlee.dev/python/api/next
1.11https://crawlee.dev/python/api
0.6https://crawlee.dev/python/api/0.6
Get startedhttps://crawlee.dev/python/docs/quick-start
Crawlee for Pythonhttps://crawlee.dev/python/
Overviewhttps://crawlee.dev/python/api
Autoscalinghttps://crawlee.dev/python/api/class/AutoscaledPool
Browser managementhttps://crawlee.dev/python/api/class/BrowserController
Configurationhttps://crawlee.dev/python/api/class/ConcurrencySettings
Crawlershttps://crawlee.dev/python/api/class/AbstractHttpCrawler
Crawling contextshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlingContext
AdaptivePlaywrightCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlingContext
AdaptivePlaywrightPostNavCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightPostNavCrawlingContext
AdaptivePlaywrightPreNavCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightPreNavCrawlingContext
BasicCrawlingContexthttps://crawlee.dev/python/api/class/BasicCrawlingContext
BeautifulSoupCrawlingContexthttps://crawlee.dev/python/api/class/BeautifulSoupCrawlingContext
FileDownloadCrawlingContexthttps://crawlee.dev/python/api/class/FileDownloadCrawlingContext
HttpCrawlingContexthttps://crawlee.dev/python/api/class/HttpCrawlingContext
HttpCrawlingResulthttps://crawlee.dev/python/api/class/HttpCrawlingResult
ParsedHttpCrawlingContexthttps://crawlee.dev/python/api/class/ParsedHttpCrawlingContext
ParselCrawlingContexthttps://crawlee.dev/python/api/class/ParselCrawlingContext
PlaywrightCrawlingContexthttps://crawlee.dev/python/api/class/PlaywrightCrawlingContext
PlaywrightPostNavCrawlingContexthttps://crawlee.dev/python/api/class/PlaywrightPostNavCrawlingContext
PlaywrightPreNavCrawlingContexthttps://crawlee.dev/python/api/class/PlaywrightPreNavCrawlingContext
PydanticAiCrawlingContexthttps://crawlee.dev/python/api/class/PydanticAiCrawlingContext
StagehandCrawlingContexthttps://crawlee.dev/python/api/class/StagehandCrawlingContext
StagehandPostNavCrawlingContexthttps://crawlee.dev/python/api/class/StagehandPostNavCrawlingContext
StagehandPreNavCrawlingContexthttps://crawlee.dev/python/api/class/StagehandPreNavCrawlingContext
Errorshttps://crawlee.dev/python/api/class/ContextPipelineFinalizationError
Event datahttps://crawlee.dev/python/api/enum/Event
Event managershttps://crawlee.dev/python/api/class/EventManager
Functionshttps://crawlee.dev/python/api/class/AddRequestsFunction
HTTP clientshttps://crawlee.dev/python/api/class/CurlImpersonateHttpClient
HTTP parsershttps://crawlee.dev/python/api/class/AbstractHttpParser
Request loadershttps://crawlee.dev/python/api/class/RequestList
Session managementhttps://crawlee.dev/python/api/class/CookieParam
Statisticshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlerStatisticState
Storage clientshttps://crawlee.dev/python/api/class/FileSystemStorageClient
Storage datahttps://crawlee.dev/python/api/class/AddRequestsResponse
Storageshttps://crawlee.dev/python/api/class/Dataset
Otherhttps://crawlee.dev/python/api/class/BasePydanticAiHtmlDistiller
https://crawlee.dev/python/
HttpCrawlingResulthttps://crawlee.dev/python/api/class/HttpCrawlingResult
BasicCrawlingContexthttps://crawlee.dev/python/api/class/BasicCrawlingContext
ParsedHttpCrawlingContexthttps://crawlee.dev/python/api/class/ParsedHttpCrawlingContext
AdaptivePlaywrightPostNavCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightPostNavCrawlingContext
FileDownloadCrawlingContexthttps://crawlee.dev/python/api/class/FileDownloadCrawlingContext
https://crawlee.dev/python/api/class/HttpCrawlingContext#Index
__hash__https://crawlee.dev/python/api/class/HttpCrawlingContext#__hash__
create_modified_copyhttps://crawlee.dev/python/api/class/HttpCrawlingContext#create_modified_copy
from_basic_crawling_contexthttps://crawlee.dev/python/api/class/HttpCrawlingContext#from_basic_crawling_context
get_snapshothttps://crawlee.dev/python/api/class/HttpCrawlingContext#get_snapshot
add_requestshttps://crawlee.dev/python/api/class/HttpCrawlingContext#add_requests
get_key_value_storehttps://crawlee.dev/python/api/class/HttpCrawlingContext#get_key_value_store
http_responsehttps://crawlee.dev/python/api/class/HttpCrawlingContext#http_response
loghttps://crawlee.dev/python/api/class/HttpCrawlingContext#log
proxy_infohttps://crawlee.dev/python/api/class/HttpCrawlingContext#proxy_info
push_datahttps://crawlee.dev/python/api/class/HttpCrawlingContext#push_data
register_deferred_cleanuphttps://crawlee.dev/python/api/class/HttpCrawlingContext#register_deferred_cleanup
requesthttps://crawlee.dev/python/api/class/HttpCrawlingContext#request
send_requesthttps://crawlee.dev/python/api/class/HttpCrawlingContext#send_request
sessionhttps://crawlee.dev/python/api/class/HttpCrawlingContext#session
use_statehttps://crawlee.dev/python/api/class/HttpCrawlingContext#use_state
https://crawlee.dev/python/api/class/HttpCrawlingContext#Methods
https://crawlee.dev/python/api/class/HttpCrawlingContext#__hash__
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L677
BasicCrawlingContext.__hash__https://crawlee.dev/python/api/class/BasicCrawlingContext#__hash__
https://crawlee.dev/python/api/class/HttpCrawlingContext#create_modified_copy
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L681
BasicCrawlingContext.create_modified_copyhttps://crawlee.dev/python/api/class/BasicCrawlingContext#create_modified_copy
https://crawlee.dev/python/api/class/HttpCrawlingContext#from_basic_crawling_context
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/crawlers/_abstract_http/_http_crawling_context.py#L22
BasicCrawlingContexthttps://crawlee.dev/python/api/class/BasicCrawlingContext
HttpResponsehttps://crawlee.dev/python/api/class/HttpResponse
https://crawlee.dev/python/api/class/HttpCrawlingContext#get_snapshot
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/crawlers/_abstract_http/_http_crawling_context.py#L27
PageSnapshothttps://crawlee.dev/python/api/class/PageSnapshot
HttpCrawlingContext.get_snapshothttps://crawlee.dev/python/api/class/HttpCrawlingContext#get_snapshot
PageSnapshothttps://crawlee.dev/python/api/class/PageSnapshot
https://crawlee.dev/python/api/class/HttpCrawlingContext#Properties
https://crawlee.dev/python/api/class/HttpCrawlingContext#add_requests
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L655
AddRequestsFunctionhttps://crawlee.dev/python/api/class/AddRequestsFunction
BasicCrawlingContext.add_requestshttps://crawlee.dev/python/api/class/BasicCrawlingContext#add_requests
https://crawlee.dev/python/api/class/HttpCrawlingContext#get_key_value_store
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L664
GetKeyValueStoreFromRequestHandlerFunctionhttps://crawlee.dev/python/api/class/GetKeyValueStoreFromRequestHandlerFunction
BasicCrawlingContext.get_key_value_storehttps://crawlee.dev/python/api/class/BasicCrawlingContext#get_key_value_store
https://crawlee.dev/python/api/class/HttpCrawlingContext#http_response
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/http_clients/_base.py#L70
HttpResponsehttps://crawlee.dev/python/api/class/HttpResponse
HttpCrawlingResult.http_responsehttps://crawlee.dev/python/api/class/HttpCrawlingResult#http_response
https://crawlee.dev/python/api/class/HttpCrawlingContext#log
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L667
BasicCrawlingContext.loghttps://crawlee.dev/python/api/class/BasicCrawlingContext#log
https://crawlee.dev/python/api/class/HttpCrawlingContext#proxy_info
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L649
BasicCrawlingContext.proxy_infohttps://crawlee.dev/python/api/class/BasicCrawlingContext#proxy_info
https://crawlee.dev/python/api/class/HttpCrawlingContext#push_data
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L658
PushDataFunctionhttps://crawlee.dev/python/api/class/PushDataFunction
BasicCrawlingContext.push_datahttps://crawlee.dev/python/api/class/BasicCrawlingContext#push_data
https://crawlee.dev/python/api/class/HttpCrawlingContext#register_deferred_cleanup
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L670
BasicCrawlingContext.register_deferred_cleanuphttps://crawlee.dev/python/api/class/BasicCrawlingContext#register_deferred_cleanup
https://crawlee.dev/python/api/class/HttpCrawlingContext#request
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L643
Requesthttps://crawlee.dev/python/api/class/Request
BasicCrawlingContext.requesthttps://crawlee.dev/python/api/class/BasicCrawlingContext#request
https://crawlee.dev/python/api/class/HttpCrawlingContext#send_request
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L652
SendRequestFunctionhttps://crawlee.dev/python/api/class/SendRequestFunction
BasicCrawlingContext.send_requesthttps://crawlee.dev/python/api/class/BasicCrawlingContext#send_request
https://crawlee.dev/python/api/class/HttpCrawlingContext#session
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L646
BasicCrawlingContext.sessionhttps://crawlee.dev/python/api/class/BasicCrawlingContext#session
https://crawlee.dev/python/api/class/HttpCrawlingContext#use_state
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/_types.py#L661
UseStateFunctionhttps://crawlee.dev/python/api/class/UseStateFunction
BasicCrawlingContext.use_statehttps://crawlee.dev/python/api/class/BasicCrawlingContext#use_state
PreviousFileDownloadCrawlingContexthttps://crawlee.dev/python/api/class/FileDownloadCrawlingContext
NextHttpCrawlingResulthttps://crawlee.dev/python/api/class/HttpCrawlingResult
__hash__https://crawlee.dev/python/api/class/HttpCrawlingContext#__hash__
create_modified_copyhttps://crawlee.dev/python/api/class/HttpCrawlingContext#create_modified_copy
from_basic_crawling_contexthttps://crawlee.dev/python/api/class/HttpCrawlingContext#from_basic_crawling_context
get_snapshothttps://crawlee.dev/python/api/class/HttpCrawlingContext#get_snapshot
add_requestshttps://crawlee.dev/python/api/class/HttpCrawlingContext#add_requests
get_key_value_storehttps://crawlee.dev/python/api/class/HttpCrawlingContext#get_key_value_store
http_responsehttps://crawlee.dev/python/api/class/HttpCrawlingContext#http_response
loghttps://crawlee.dev/python/api/class/HttpCrawlingContext#log
proxy_infohttps://crawlee.dev/python/api/class/HttpCrawlingContext#proxy_info
push_datahttps://crawlee.dev/python/api/class/HttpCrawlingContext#push_data
register_deferred_cleanuphttps://crawlee.dev/python/api/class/HttpCrawlingContext#register_deferred_cleanup
requesthttps://crawlee.dev/python/api/class/HttpCrawlingContext#request
send_requesthttps://crawlee.dev/python/api/class/HttpCrawlingContext#send_request
sessionhttps://crawlee.dev/python/api/class/HttpCrawlingContext#session
use_statehttps://crawlee.dev/python/api/class/HttpCrawlingContext#use_state
https://crawlee.dev
Quick starthttps://crawlee.dev/python/docs/quick-start
Guideshttps://crawlee.dev/python/docs/guides
API referencehttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Discordhttps://discord.com/invite/jyEM2PRvMU
Stack Overflowhttps://stackoverflow.com/questions/tagged/crawlee-python
Twitterhttps://twitter.com/apify
YouTubehttps://www.youtube.com/apify
Apify platformhttps://apify.com
Docusaurushttps://docusaurus.io
GitHubhttps://github.com/apify/crawlee-python

Viewport: width=device-width, initial-scale=1.0


URLs of crawlers that visited me.