René's URL Explorer Experiment


Title: crawlee | API | Crawlee for Python · Fast, reliable Python web crawlers.

Open Graph Title: crawlee | API | Crawlee for Python · Fast, reliable Python web crawlers.

Description: Crawlee helps you build and maintain your Python crawlers. It's open source and modern, with type hints for Python to help you catch bugs early.

Open Graph Description: Crawlee helps you build and maintain your Python crawlers. It's open source and modern, with type hints for Python to help you catch bugs early.

Opengraph URL: https://crawlee.dev/python/api

Generator: Docusaurus v3.10.2

direct link

Domain: crawlee.dev


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[]}

twitter:cardsummary_large_image
og:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
twitter:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
og:localeen
docusaurus_localeen
docsearch:languageen
docusaurus_version1.11
docusaurus_tagdocs-default-1.11
docsearch:version1.11
docsearch:docusaurus_tagdocs-default-1.11

Links:

Skip to main contenthttps://crawlee.dev/python/api#__docusaurus_skipToContent_fallback
Crawlee for Pythonhttps://crawlee.dev/python/
https://crawlee.dev/js
https://crawlee.dev/python/
https://crawlee.dev
Docshttps://crawlee.dev/python/docs/quick-start
APIhttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Bloghttps://crawlee.dev/blog
1.11https://crawlee.dev/python/api
Nexthttps://crawlee.dev/python/api/next
1.11https://crawlee.dev/python/api
0.6https://crawlee.dev/python/api/0.6
Get startedhttps://crawlee.dev/python/docs/quick-start
Crawlee for Pythonhttps://crawlee.dev/python/
Overviewhttps://crawlee.dev/python/api
Autoscalinghttps://crawlee.dev/python/api/class/AutoscaledPool
Browser managementhttps://crawlee.dev/python/api/class/BrowserController
Configurationhttps://crawlee.dev/python/api/class/ConcurrencySettings
Crawlershttps://crawlee.dev/python/api/class/AbstractHttpCrawler
Crawling contextshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlingContext
Errorshttps://crawlee.dev/python/api/class/ContextPipelineFinalizationError
Event datahttps://crawlee.dev/python/api/enum/Event
Event managershttps://crawlee.dev/python/api/class/EventManager
Functionshttps://crawlee.dev/python/api/class/AddRequestsFunction
HTTP clientshttps://crawlee.dev/python/api/class/CurlImpersonateHttpClient
HTTP parsershttps://crawlee.dev/python/api/class/AbstractHttpParser
Request loadershttps://crawlee.dev/python/api/class/RequestList
Session managementhttps://crawlee.dev/python/api/class/CookieParam
Statisticshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlerStatisticState
Storage clientshttps://crawlee.dev/python/api/class/FileSystemStorageClient
Storage datahttps://crawlee.dev/python/api/class/AddRequestsResponse
Storageshttps://crawlee.dev/python/api/class/Dataset
Otherhttps://crawlee.dev/python/api/class/BasePydanticAiHtmlDistiller
https://crawlee.dev/python/
https://crawlee.dev/python/api#Index
AutoscaledPoolhttps://crawlee.dev/python/api/class/AutoscaledPool
Snapshotterhttps://crawlee.dev/python/api/class/Snapshotter
SystemStatushttps://crawlee.dev/python/api/class/SystemStatus
BrowserControllerhttps://crawlee.dev/python/api/class/BrowserController
BrowserPluginhttps://crawlee.dev/python/api/class/BrowserPlugin
BrowserPoolhttps://crawlee.dev/python/api/class/BrowserPool
PlaywrightBrowserControllerhttps://crawlee.dev/python/api/class/PlaywrightBrowserController
PlaywrightBrowserPluginhttps://crawlee.dev/python/api/class/PlaywrightBrowserPlugin
PlaywrightPersistentBrowserhttps://crawlee.dev/python/api/class/PlaywrightPersistentBrowser
StagehandBrowserControllerhttps://crawlee.dev/python/api/class/StagehandBrowserController
StagehandBrowserPluginhttps://crawlee.dev/python/api/class/StagehandBrowserPlugin
StagehandOptionshttps://crawlee.dev/python/api/class/StagehandOptions
StagehandPagehttps://crawlee.dev/python/api/class/StagehandPage
ConcurrencySettingshttps://crawlee.dev/python/api/class/ConcurrencySettings
Configurationhttps://crawlee.dev/python/api/class/Configuration
ProxyConfigurationhttps://crawlee.dev/python/api/class/ProxyConfiguration
ServiceLocatorhttps://crawlee.dev/python/api/class/ServiceLocator
AbstractHttpCrawlerhttps://crawlee.dev/python/api/class/AbstractHttpCrawler
AdaptivePlaywrightCrawlerhttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawler
BasicCrawlerhttps://crawlee.dev/python/api/class/BasicCrawler
BeautifulSoupCrawlerhttps://crawlee.dev/python/api/class/BeautifulSoupCrawler
FileDownloadCrawlerhttps://crawlee.dev/python/api/class/FileDownloadCrawler
HttpCrawlerhttps://crawlee.dev/python/api/class/HttpCrawler
ParselCrawlerhttps://crawlee.dev/python/api/class/ParselCrawler
PlaywrightCrawlerhttps://crawlee.dev/python/api/class/PlaywrightCrawler
PydanticAiCrawlerhttps://crawlee.dev/python/api/class/PydanticAiCrawler
StagehandCrawlerhttps://crawlee.dev/python/api/class/StagehandCrawler
AdaptivePlaywrightCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlingContext
AdaptivePlaywrightPostNavCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightPostNavCrawlingContext
AdaptivePlaywrightPreNavCrawlingContexthttps://crawlee.dev/python/api/class/AdaptivePlaywrightPreNavCrawlingContext
BasicCrawlingContexthttps://crawlee.dev/python/api/class/BasicCrawlingContext
BeautifulSoupCrawlingContexthttps://crawlee.dev/python/api/class/BeautifulSoupCrawlingContext
FileDownloadCrawlingContexthttps://crawlee.dev/python/api/class/FileDownloadCrawlingContext
HttpCrawlingContexthttps://crawlee.dev/python/api/class/HttpCrawlingContext
HttpCrawlingResulthttps://crawlee.dev/python/api/class/HttpCrawlingResult
ParsedHttpCrawlingContexthttps://crawlee.dev/python/api/class/ParsedHttpCrawlingContext
ParselCrawlingContexthttps://crawlee.dev/python/api/class/ParselCrawlingContext
PlaywrightCrawlingContexthttps://crawlee.dev/python/api/class/PlaywrightCrawlingContext
PlaywrightPostNavCrawlingContexthttps://crawlee.dev/python/api/class/PlaywrightPostNavCrawlingContext
PlaywrightPreNavCrawlingContexthttps://crawlee.dev/python/api/class/PlaywrightPreNavCrawlingContext
PydanticAiCrawlingContexthttps://crawlee.dev/python/api/class/PydanticAiCrawlingContext
StagehandCrawlingContexthttps://crawlee.dev/python/api/class/StagehandCrawlingContext
StagehandPostNavCrawlingContexthttps://crawlee.dev/python/api/class/StagehandPostNavCrawlingContext
StagehandPreNavCrawlingContexthttps://crawlee.dev/python/api/class/StagehandPreNavCrawlingContext
ContextPipelineFinalizationErrorhttps://crawlee.dev/python/api/class/ContextPipelineFinalizationError
ContextPipelineInitializationErrorhttps://crawlee.dev/python/api/class/ContextPipelineInitializationError
ContextPipelineInterruptedErrorhttps://crawlee.dev/python/api/class/ContextPipelineInterruptedError
HttpClientStatusCodeErrorhttps://crawlee.dev/python/api/class/HttpClientStatusCodeError
HttpStatusCodeErrorhttps://crawlee.dev/python/api/class/HttpStatusCodeError
PersistentRateLimitErrorhttps://crawlee.dev/python/api/class/PersistentRateLimitError
ProxyErrorhttps://crawlee.dev/python/api/class/ProxyError
RequestCollisionErrorhttps://crawlee.dev/python/api/class/RequestCollisionError
RequestHandlerErrorhttps://crawlee.dev/python/api/class/RequestHandlerError
RequestThrottledErrorhttps://crawlee.dev/python/api/class/RequestThrottledError
ServiceConflictErrorhttps://crawlee.dev/python/api/class/ServiceConflictError
SessionErrorhttps://crawlee.dev/python/api/class/SessionError
UserDefinedErrorHandlerErrorhttps://crawlee.dev/python/api/class/UserDefinedErrorHandlerError
Eventhttps://crawlee.dev/python/api/enum/Event
EventAbortingDatahttps://crawlee.dev/python/api/class/EventAbortingData
EventCrawlerStatusDatahttps://crawlee.dev/python/api/class/EventCrawlerStatusData
EventExitDatahttps://crawlee.dev/python/api/class/EventExitData
EventMigratingDatahttps://crawlee.dev/python/api/class/EventMigratingData
EventPersistStateDatahttps://crawlee.dev/python/api/class/EventPersistStateData
EventSystemInfoDatahttps://crawlee.dev/python/api/class/EventSystemInfoData
EventManagerhttps://crawlee.dev/python/api/class/EventManager
LocalEventManagerhttps://crawlee.dev/python/api/class/LocalEventManager
AddRequestsFunctionhttps://crawlee.dev/python/api/class/AddRequestsFunction
BlockRequestsFunctionhttps://crawlee.dev/python/api/class/BlockRequestsFunction
EnqueueLinksFunctionhttps://crawlee.dev/python/api/class/EnqueueLinksFunction
ExtractFunctionhttps://crawlee.dev/python/api/class/ExtractFunction
ExtractLinksFunctionhttps://crawlee.dev/python/api/class/ExtractLinksFunction
GetKeyValueStoreFunctionhttps://crawlee.dev/python/api/class/GetKeyValueStoreFunction
PushDataFunctionhttps://crawlee.dev/python/api/class/PushDataFunction
PydanticAiHtmlDistillerhttps://crawlee.dev/python/api/class/PydanticAiHtmlDistiller
SendRequestFunctionhttps://crawlee.dev/python/api/class/SendRequestFunction
UseStateFunctionhttps://crawlee.dev/python/api/class/UseStateFunction
CurlImpersonateHttpClienthttps://crawlee.dev/python/api/class/CurlImpersonateHttpClient
HttpClienthttps://crawlee.dev/python/api/class/HttpClient
HttpxHttpClienthttps://crawlee.dev/python/api/class/HttpxHttpClient
ImpitHttpClienthttps://crawlee.dev/python/api/class/ImpitHttpClient
AbstractHttpParserhttps://crawlee.dev/python/api/class/AbstractHttpParser
BeautifulSoupParserhttps://crawlee.dev/python/api/class/BeautifulSoupParser
NoParserhttps://crawlee.dev/python/api/class/NoParser
ParselParserhttps://crawlee.dev/python/api/class/ParselParser
BasePydanticAiHtmlDistillerhttps://crawlee.dev/python/api/class/BasePydanticAiHtmlDistiller
BasePydanticAiHtmlExtractorhttps://crawlee.dev/python/api/class/BasePydanticAiHtmlExtractor
BrowserforgeFingerprintGeneratorhttps://crawlee.dev/python/api/class/BrowserforgeFingerprintGenerator
ContextPipelinehttps://crawlee.dev/python/api/class/ContextPipeline
CrawlerInstrumentorhttps://crawlee.dev/python/api/class/CrawlerInstrumentor
DefaultRenderingTypePredictorhttps://crawlee.dev/python/api/class/DefaultRenderingTypePredictor
FingerprintGeneratorhttps://crawlee.dev/python/api/class/FingerprintGenerator
HeaderGeneratorhttps://crawlee.dev/python/api/class/HeaderGenerator
HttpHeadershttps://crawlee.dev/python/api/class/HttpHeaders
HttpResponsehttps://crawlee.dev/python/api/class/HttpResponse
PageSnapshothttps://crawlee.dev/python/api/class/PageSnapshot
ProxyInfohttps://crawlee.dev/python/api/class/ProxyInfo
PydanticAiCleanHtmlDistillerhttps://crawlee.dev/python/api/class/PydanticAiCleanHtmlDistiller
PydanticAiDirectExtractorhttps://crawlee.dev/python/api/class/PydanticAiDirectExtractor
PydanticAiHtmlExtractorhttps://crawlee.dev/python/api/class/PydanticAiHtmlExtractor
PydanticAiSelectorExtractorhttps://crawlee.dev/python/api/class/PydanticAiSelectorExtractor
PydanticAiSkeletonDistillerhttps://crawlee.dev/python/api/class/PydanticAiSkeletonDistiller
PydanticAiUsageStatshttps://crawlee.dev/python/api/class/PydanticAiUsageStats
RenderingTypePredictionhttps://crawlee.dev/python/api/class/RenderingTypePrediction
RenderingTypePredictorhttps://crawlee.dev/python/api/class/RenderingTypePredictor
RequestOptionshttps://crawlee.dev/python/api/class/RequestOptions
Routerhttps://crawlee.dev/python/api/class/Router
RequestListhttps://crawlee.dev/python/api/class/RequestList
RequestLoaderhttps://crawlee.dev/python/api/class/RequestLoader
RequestManagerhttps://crawlee.dev/python/api/class/RequestManager
RequestManagerTandemhttps://crawlee.dev/python/api/class/RequestManagerTandem
SitemapRequestLoaderhttps://crawlee.dev/python/api/class/SitemapRequestLoader
ThrottlingRequestManagerhttps://crawlee.dev/python/api/class/ThrottlingRequestManager
CookieParamhttps://crawlee.dev/python/api/class/CookieParam
Sessionhttps://crawlee.dev/python/api/class/Session
SessionCookieshttps://crawlee.dev/python/api/class/SessionCookies
SessionPoolhttps://crawlee.dev/python/api/class/SessionPool
AdaptivePlaywrightCrawlerStatisticStatehttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlerStatisticState
FinalStatisticshttps://crawlee.dev/python/api/class/FinalStatistics
Statisticshttps://crawlee.dev/python/api/class/Statistics
StatisticsStatehttps://crawlee.dev/python/api/class/StatisticsState
FileSystemStorageClienthttps://crawlee.dev/python/api/class/FileSystemStorageClient
MemoryStorageClienthttps://crawlee.dev/python/api/class/MemoryStorageClient
RedisStorageClienthttps://crawlee.dev/python/api/class/RedisStorageClient
SqlStorageClienthttps://crawlee.dev/python/api/class/SqlStorageClient
StorageClienthttps://crawlee.dev/python/api/class/StorageClient
AddRequestsResponsehttps://crawlee.dev/python/api/class/AddRequestsResponse
DatasetItemsListPagehttps://crawlee.dev/python/api/class/DatasetItemsListPage
DatasetMetadatahttps://crawlee.dev/python/api/class/DatasetMetadata
KeyValueStoreMetadatahttps://crawlee.dev/python/api/class/KeyValueStoreMetadata
KeyValueStoreRecordhttps://crawlee.dev/python/api/class/KeyValueStoreRecord
KeyValueStoreRecordMetadatahttps://crawlee.dev/python/api/class/KeyValueStoreRecordMetadata
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
Requesthttps://crawlee.dev/python/api/class/Request
RequestQueueMetadatahttps://crawlee.dev/python/api/class/RequestQueueMetadata
StorageMetadatahttps://crawlee.dev/python/api/class/StorageMetadata
UnprocessedRequesthttps://crawlee.dev/python/api/class/UnprocessedRequest
Datasethttps://crawlee.dev/python/api/class/Dataset
KeyValueStorehttps://crawlee.dev/python/api/class/KeyValueStore
RequestQueuehttps://crawlee.dev/python/api/class/RequestQueue
Storagehttps://crawlee.dev/python/api/class/Storage
https://crawlee.dev
Quick starthttps://crawlee.dev/python/docs/quick-start
Guideshttps://crawlee.dev/python/docs/guides
API referencehttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Discordhttps://discord.com/invite/jyEM2PRvMU
Stack Overflowhttps://stackoverflow.com/questions/tagged/crawlee-python
Twitterhttps://twitter.com/apify
YouTubehttps://www.youtube.com/apify
Apify platformhttps://apify.com
Docusaurushttps://docusaurus.io
GitHubhttps://github.com/apify/crawlee-python

Viewport: width=device-width, initial-scale=1.0


URLs of crawlers that visited me.