René's URL Explorer Experiment


Title: RequestQueue | API | Crawlee for Python · Fast, reliable Python web crawlers.

Open Graph Title: RequestQueue | API | Crawlee for Python · Fast, reliable Python web crawlers.

Description: Request queue is a storage for managing HTTP requests.

Open Graph Description: Request queue is a storage for managing HTTP requests.

Opengraph URL: https://crawlee.dev/python/api/class/RequestQueue

Generator: Docusaurus v3.10.2

direct link

Domain: crawlee.dev


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[]}

twitter:cardsummary_large_image
og:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
twitter:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
og:localeen
docusaurus_localeen
docsearch:languageen
docusaurus_version1.11
docusaurus_tagdocs-default-1.11
docsearch:version1.11
docsearch:docusaurus_tagdocs-default-1.11

Links:

Skip to main contenthttps://crawlee.dev/python/api/class/RequestQueue#__docusaurus_skipToContent_fallback
Crawlee for Pythonhttps://crawlee.dev/python/
https://crawlee.dev/js
https://crawlee.dev/python/
https://crawlee.dev
Docshttps://crawlee.dev/python/docs/quick-start
APIhttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Bloghttps://crawlee.dev/blog
1.11https://crawlee.dev/python/api
Nexthttps://crawlee.dev/python/api/next
1.11https://crawlee.dev/python/api
0.6https://crawlee.dev/python/api/0.6
Get startedhttps://crawlee.dev/python/docs/quick-start
Crawlee for Pythonhttps://crawlee.dev/python/
Overviewhttps://crawlee.dev/python/api
Autoscalinghttps://crawlee.dev/python/api/class/AutoscaledPool
Browser managementhttps://crawlee.dev/python/api/class/BrowserController
Configurationhttps://crawlee.dev/python/api/class/ConcurrencySettings
Crawlershttps://crawlee.dev/python/api/class/AbstractHttpCrawler
Crawling contextshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlingContext
Errorshttps://crawlee.dev/python/api/class/ContextPipelineFinalizationError
Event datahttps://crawlee.dev/python/api/enum/Event
Event managershttps://crawlee.dev/python/api/class/EventManager
Functionshttps://crawlee.dev/python/api/class/AddRequestsFunction
HTTP clientshttps://crawlee.dev/python/api/class/CurlImpersonateHttpClient
HTTP parsershttps://crawlee.dev/python/api/class/AbstractHttpParser
Request loadershttps://crawlee.dev/python/api/class/RequestList
Session managementhttps://crawlee.dev/python/api/class/CookieParam
Statisticshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlerStatisticState
Storage clientshttps://crawlee.dev/python/api/class/FileSystemStorageClient
Storage datahttps://crawlee.dev/python/api/class/AddRequestsResponse
Storageshttps://crawlee.dev/python/api/class/Dataset
Datasethttps://crawlee.dev/python/api/class/Dataset
KeyValueStorehttps://crawlee.dev/python/api/class/KeyValueStore
RequestQueuehttps://crawlee.dev/python/api/class/RequestQueue
Storagehttps://crawlee.dev/python/api/class/Storage
Otherhttps://crawlee.dev/python/api/class/BasePydanticAiHtmlDistiller
https://crawlee.dev/python/
Storagehttps://crawlee.dev/python/api/class/Storage
RequestManagerhttps://crawlee.dev/python/api/class/RequestManager
https://crawlee.dev/python/api/class/RequestQueue#Index
__init__https://crawlee.dev/python/api/class/RequestQueue#__init__
add_requesthttps://crawlee.dev/python/api/class/RequestQueue#add_request
add_requestshttps://crawlee.dev/python/api/class/RequestQueue#add_requests
drophttps://crawlee.dev/python/api/class/RequestQueue#drop
fetch_next_requesthttps://crawlee.dev/python/api/class/RequestQueue#fetch_next_request
get_handled_counthttps://crawlee.dev/python/api/class/RequestQueue#get_handled_count
get_metadatahttps://crawlee.dev/python/api/class/RequestQueue#get_metadata
get_requesthttps://crawlee.dev/python/api/class/RequestQueue#get_request
get_total_counthttps://crawlee.dev/python/api/class/RequestQueue#get_total_count
is_emptyhttps://crawlee.dev/python/api/class/RequestQueue#is_empty
is_finishedhttps://crawlee.dev/python/api/class/RequestQueue#is_finished
mark_request_as_handledhttps://crawlee.dev/python/api/class/RequestQueue#mark_request_as_handled
openhttps://crawlee.dev/python/api/class/RequestQueue#open
purgehttps://crawlee.dev/python/api/class/RequestQueue#purge
reclaim_requesthttps://crawlee.dev/python/api/class/RequestQueue#reclaim_request
to_tandemhttps://crawlee.dev/python/api/class/RequestQueue#to_tandem
idhttps://crawlee.dev/python/api/class/RequestQueue#id
namehttps://crawlee.dev/python/api/class/RequestQueue#name
https://crawlee.dev/python/api/class/RequestQueue#Methods
https://crawlee.dev/python/api/class/RequestQueue#__init__
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L75
RequestQueueClienthttps://crawlee.dev/python/api/class/RequestQueueClient
https://crawlee.dev/python/api/class/RequestQueue#add_request
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L178
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
RequestManager.add_requesthttps://crawlee.dev/python/api/class/RequestManager#add_request
Requesthttps://crawlee.dev/python/api/class/Request
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
https://crawlee.dev/python/api/class/RequestQueue#add_requests
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L201
RequestManager.add_requestshttps://crawlee.dev/python/api/class/RequestManager#add_requests
Requesthttps://crawlee.dev/python/api/class/Request
https://crawlee.dev/python/api/class/RequestQueue#drop
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L145
RequestManager.drophttps://crawlee.dev/python/api/class/RequestManager#drop
https://crawlee.dev/python/api/class/RequestQueue#fetch_next_request
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L251
Requesthttps://crawlee.dev/python/api/class/Request
RequestManager.fetch_next_requesthttps://crawlee.dev/python/api/class/RequestManager#fetch_next_request
Requesthttps://crawlee.dev/python/api/class/Request
https://crawlee.dev/python/api/class/RequestQueue#get_handled_count
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L109
RequestManager.get_handled_counthttps://crawlee.dev/python/api/class/RequestManager#get_handled_count
https://crawlee.dev/python/api/class/RequestQueue#get_metadata
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L105
DatasetMetadatahttps://crawlee.dev/python/api/class/DatasetMetadata
KeyValueStoreMetadatahttps://crawlee.dev/python/api/class/KeyValueStoreMetadata
RequestQueueMetadatahttps://crawlee.dev/python/api/class/RequestQueueMetadata
Storage.get_metadatahttps://crawlee.dev/python/api/class/Storage#get_metadata
DatasetMetadatahttps://crawlee.dev/python/api/class/DatasetMetadata
KeyValueStoreMetadatahttps://crawlee.dev/python/api/class/KeyValueStoreMetadata
RequestQueueMetadatahttps://crawlee.dev/python/api/class/RequestQueueMetadata
https://crawlee.dev/python/api/class/RequestQueue#get_request
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L268
Requesthttps://crawlee.dev/python/api/class/Request
Requesthttps://crawlee.dev/python/api/class/Request
https://crawlee.dev/python/api/class/RequestQueue#get_total_count
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L114
RequestManager.get_total_counthttps://crawlee.dev/python/api/class/RequestManager#get_total_count
https://crawlee.dev/python/api/class/RequestQueue#is_empty
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L316
RequestManager.is_emptyhttps://crawlee.dev/python/api/class/RequestManager#is_empty
https://crawlee.dev/python/api/class/RequestQueue#is_finished
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L328
RequestManager.is_finishedhttps://crawlee.dev/python/api/class/RequestManager#is_finished
https://crawlee.dev/python/api/class/RequestQueue#mark_request_as_handled
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L279
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
RequestManager.mark_request_as_handledhttps://crawlee.dev/python/api/class/RequestManager#mark_request_as_handled
Requesthttps://crawlee.dev/python/api/class/Request
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
https://crawlee.dev/python/api/class/RequestQueue#open
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L120
Storagehttps://crawlee.dev/python/api/class/Storage
Storage.openhttps://crawlee.dev/python/api/class/Storage#open
Configurationhttps://crawlee.dev/python/api/class/Configuration
StorageClienthttps://crawlee.dev/python/api/class/StorageClient
Storagehttps://crawlee.dev/python/api/class/Storage
https://crawlee.dev/python/api/class/RequestQueue#purge
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L153
RequestManager.purgehttps://crawlee.dev/python/api/class/RequestManager#purge
https://crawlee.dev/python/api/class/RequestQueue#reclaim_request
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L294
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
RequestManager.reclaim_requesthttps://crawlee.dev/python/api/class/RequestManager#reclaim_request
Requesthttps://crawlee.dev/python/api/class/Request
ProcessedRequesthttps://crawlee.dev/python/api/class/ProcessedRequest
https://crawlee.dev/python/api/class/RequestQueue#to_tandem
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/request_loaders/_request_loader.py#L56
RequestManagerTandemhttps://crawlee.dev/python/api/class/RequestManagerTandem
RequestLoader.to_tandemhttps://crawlee.dev/python/api/class/RequestLoader#to_tandem
RequestManagerTandemhttps://crawlee.dev/python/api/class/RequestManagerTandem
https://crawlee.dev/python/api/class/RequestQueue#Properties
https://crawlee.dev/python/api/class/RequestQueue#id
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L96
Storage.idhttps://crawlee.dev/python/api/class/Storage#id
https://crawlee.dev/python/api/class/RequestQueue#name
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/storages/_request_queue.py#L101
Storage.namehttps://crawlee.dev/python/api/class/Storage#name
PreviousKeyValueStorehttps://crawlee.dev/python/api/class/KeyValueStore
NextStoragehttps://crawlee.dev/python/api/class/Storage
__init__https://crawlee.dev/python/api/class/RequestQueue#__init__
add_requesthttps://crawlee.dev/python/api/class/RequestQueue#add_request
add_requestshttps://crawlee.dev/python/api/class/RequestQueue#add_requests
drophttps://crawlee.dev/python/api/class/RequestQueue#drop
fetch_next_requesthttps://crawlee.dev/python/api/class/RequestQueue#fetch_next_request
get_handled_counthttps://crawlee.dev/python/api/class/RequestQueue#get_handled_count
get_metadatahttps://crawlee.dev/python/api/class/RequestQueue#get_metadata
get_requesthttps://crawlee.dev/python/api/class/RequestQueue#get_request
get_total_counthttps://crawlee.dev/python/api/class/RequestQueue#get_total_count
is_emptyhttps://crawlee.dev/python/api/class/RequestQueue#is_empty
is_finishedhttps://crawlee.dev/python/api/class/RequestQueue#is_finished
mark_request_as_handledhttps://crawlee.dev/python/api/class/RequestQueue#mark_request_as_handled
openhttps://crawlee.dev/python/api/class/RequestQueue#open
purgehttps://crawlee.dev/python/api/class/RequestQueue#purge
reclaim_requesthttps://crawlee.dev/python/api/class/RequestQueue#reclaim_request
to_tandemhttps://crawlee.dev/python/api/class/RequestQueue#to_tandem
idhttps://crawlee.dev/python/api/class/RequestQueue#id
namehttps://crawlee.dev/python/api/class/RequestQueue#name
https://crawlee.dev
Quick starthttps://crawlee.dev/python/docs/quick-start
Guideshttps://crawlee.dev/python/docs/guides
API referencehttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Discordhttps://discord.com/invite/jyEM2PRvMU
Stack Overflowhttps://stackoverflow.com/questions/tagged/crawlee-python
Twitterhttps://twitter.com/apify
YouTubehttps://www.youtube.com/apify
Apify platformhttps://apify.com
Docusaurushttps://docusaurus.io
GitHubhttps://github.com/apify/crawlee-python

Viewport: width=device-width, initial-scale=1.0


URLs of crawlers that visited me.