René's URL Explorer Experiment


Title: Configuration | API | Crawlee for Python · Fast, reliable Python web crawlers.

Open Graph Title: Configuration | API | Crawlee for Python · Fast, reliable Python web crawlers.

Description: Configuration settings for the Crawlee project.

Open Graph Description: Configuration settings for the Crawlee project.

Opengraph URL: https://crawlee.dev/python/api/class/Configuration

Generator: Docusaurus v3.10.2

direct link

Domain: crawlee.dev


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[]}

twitter:cardsummary_large_image
og:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
twitter:imagehttps://crawlee.dev/python/img/crawlee-python-og.png
og:localeen
docusaurus_localeen
docsearch:languageen
docusaurus_version1.11
docusaurus_tagdocs-default-1.11
docsearch:version1.11
docsearch:docusaurus_tagdocs-default-1.11

Links:

Skip to main contenthttps://crawlee.dev/python/api/class/Configuration#__docusaurus_skipToContent_fallback
Crawlee for Pythonhttps://crawlee.dev/python/
https://crawlee.dev/js
https://crawlee.dev/python/
https://crawlee.dev
Docshttps://crawlee.dev/python/docs/quick-start
APIhttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Bloghttps://crawlee.dev/blog
1.11https://crawlee.dev/python/api
Nexthttps://crawlee.dev/python/api/next
1.11https://crawlee.dev/python/api
0.6https://crawlee.dev/python/api/0.6
Get startedhttps://crawlee.dev/python/docs/quick-start
Crawlee for Pythonhttps://crawlee.dev/python/
Overviewhttps://crawlee.dev/python/api
Autoscalinghttps://crawlee.dev/python/api/class/AutoscaledPool
Browser managementhttps://crawlee.dev/python/api/class/BrowserController
Configurationhttps://crawlee.dev/python/api/class/ConcurrencySettings
ConcurrencySettingshttps://crawlee.dev/python/api/class/ConcurrencySettings
Configurationhttps://crawlee.dev/python/api/class/Configuration
ProxyConfigurationhttps://crawlee.dev/python/api/class/ProxyConfiguration
ServiceLocatorhttps://crawlee.dev/python/api/class/ServiceLocator
Crawlershttps://crawlee.dev/python/api/class/AbstractHttpCrawler
Crawling contextshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlingContext
Errorshttps://crawlee.dev/python/api/class/ContextPipelineFinalizationError
Event datahttps://crawlee.dev/python/api/enum/Event
Event managershttps://crawlee.dev/python/api/class/EventManager
Functionshttps://crawlee.dev/python/api/class/AddRequestsFunction
HTTP clientshttps://crawlee.dev/python/api/class/CurlImpersonateHttpClient
HTTP parsershttps://crawlee.dev/python/api/class/AbstractHttpParser
Request loadershttps://crawlee.dev/python/api/class/RequestList
Session managementhttps://crawlee.dev/python/api/class/CookieParam
Statisticshttps://crawlee.dev/python/api/class/AdaptivePlaywrightCrawlerStatisticState
Storage clientshttps://crawlee.dev/python/api/class/FileSystemStorageClient
Storage datahttps://crawlee.dev/python/api/class/AddRequestsResponse
Storageshttps://crawlee.dev/python/api/class/Dataset
Otherhttps://crawlee.dev/python/api/class/BasePydanticAiHtmlDistiller
https://crawlee.dev/python/
https://crawlee.dev/python/api/class/Configuration#Index
get_global_configurationhttps://crawlee.dev/python/api/class/Configuration#get_global_configuration
available_memory_ratiohttps://crawlee.dev/python/api/class/Configuration#available_memory_ratio
default_browser_pathhttps://crawlee.dev/python/api/class/Configuration#default_browser_path
disable_browser_sandboxhttps://crawlee.dev/python/api/class/Configuration#disable_browser_sandbox
headlesshttps://crawlee.dev/python/api/class/Configuration#headless
internal_timeouthttps://crawlee.dev/python/api/class/Configuration#internal_timeout
log_levelhttps://crawlee.dev/python/api/class/Configuration#log_level
max_client_errorshttps://crawlee.dev/python/api/class/Configuration#max_client_errors
max_event_loop_delayhttps://crawlee.dev/python/api/class/Configuration#max_event_loop_delay
max_used_cpu_ratiohttps://crawlee.dev/python/api/class/Configuration#max_used_cpu_ratio
max_used_memory_ratiohttps://crawlee.dev/python/api/class/Configuration#max_used_memory_ratio
memory_mbyteshttps://crawlee.dev/python/api/class/Configuration#memory_mbytes
model_confighttps://crawlee.dev/python/api/class/Configuration#model_config
persist_state_intervalhttps://crawlee.dev/python/api/class/Configuration#persist_state_interval
purge_on_starthttps://crawlee.dev/python/api/class/Configuration#purge_on_start
storage_dirhttps://crawlee.dev/python/api/class/Configuration#storage_dir
system_info_intervalhttps://crawlee.dev/python/api/class/Configuration#system_info_interval
https://crawlee.dev/python/api/class/Configuration#Methods
https://crawlee.dev/python/api/class/Configuration#get_global_configuration
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L220
https://crawlee.dev/python/api/class/Configuration#Properties
https://crawlee.dev/python/api/class/Configuration#available_memory_ratio
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L178
https://crawlee.dev/python/api/class/Configuration#default_browser_path
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L38
https://playwright.dev/docs/api/class-browsertype#browser-type-launchhttps://playwright.dev/docs/api/class-browsertype#browser-type-launch
https://crawlee.dev/python/api/class/Configuration#disable_browser_sandbox
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L53
https://playwright.dev/docs/api/class-browsertype#browser-type-launchhttps://playwright.dev/docs/api/class-browsertype#browser-type-launch
https://crawlee.dev/python/api/class/Configuration#headless
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L204
https://playwright.dev/docs/api/class-browsertype#browser-type-launchhttps://playwright.dev/docs/api/class-browsertype#browser-type-launch
https://crawlee.dev/python/api/class/Configuration#internal_timeout
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L35
https://crawlee.dev/python/api/class/Configuration#log_level
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L67
LogLevelhttps://crawlee.dev/python/api#LogLevel
https://crawlee.dev/python/api/class/Configuration#max_client_errors
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L154
https://crawlee.dev/python/api/class/Configuration#max_event_loop_delay
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L142
timedelta_mshttps://crawlee.dev/python/api#timedelta_ms
https://crawlee.dev/python/api/class/Configuration#max_used_cpu_ratio
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L114
https://crawlee.dev/python/api/class/Configuration#max_used_memory_ratio
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L128
https://crawlee.dev/python/api/class/Configuration#memory_mbytes
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L166
https://crawlee.dev/python/api/class/Configuration#model_config
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L33
https://crawlee.dev/python/api/class/Configuration#persist_state_interval
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L90
timedelta_mshttps://crawlee.dev/python/api#timedelta_ms
https://crawlee.dev/python/api/class/Configuration#purge_on_start
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L79
https://crawlee.dev/python/api/class/Configuration#storage_dir
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L193
https://crawlee.dev/python/api/class/Configuration#system_info_interval
https://github.com/apify/crawlee-python/blob/ea492fa9b5377f44082dc51d582e85d686b23c75//src/crawlee/configuration.py#L102
timedelta_mshttps://crawlee.dev/python/api#timedelta_ms
PreviousConcurrencySettingshttps://crawlee.dev/python/api/class/ConcurrencySettings
NextProxyConfigurationhttps://crawlee.dev/python/api/class/ProxyConfiguration
get_global_configurationhttps://crawlee.dev/python/api/class/Configuration#get_global_configuration
available_memory_ratiohttps://crawlee.dev/python/api/class/Configuration#available_memory_ratio
default_browser_pathhttps://crawlee.dev/python/api/class/Configuration#default_browser_path
disable_browser_sandboxhttps://crawlee.dev/python/api/class/Configuration#disable_browser_sandbox
headlesshttps://crawlee.dev/python/api/class/Configuration#headless
internal_timeouthttps://crawlee.dev/python/api/class/Configuration#internal_timeout
log_levelhttps://crawlee.dev/python/api/class/Configuration#log_level
max_client_errorshttps://crawlee.dev/python/api/class/Configuration#max_client_errors
max_event_loop_delayhttps://crawlee.dev/python/api/class/Configuration#max_event_loop_delay
max_used_cpu_ratiohttps://crawlee.dev/python/api/class/Configuration#max_used_cpu_ratio
max_used_memory_ratiohttps://crawlee.dev/python/api/class/Configuration#max_used_memory_ratio
memory_mbyteshttps://crawlee.dev/python/api/class/Configuration#memory_mbytes
model_confighttps://crawlee.dev/python/api/class/Configuration#model_config
persist_state_intervalhttps://crawlee.dev/python/api/class/Configuration#persist_state_interval
purge_on_starthttps://crawlee.dev/python/api/class/Configuration#purge_on_start
storage_dirhttps://crawlee.dev/python/api/class/Configuration#storage_dir
system_info_intervalhttps://crawlee.dev/python/api/class/Configuration#system_info_interval
https://crawlee.dev
Quick starthttps://crawlee.dev/python/docs/quick-start
Guideshttps://crawlee.dev/python/docs/guides
API referencehttps://crawlee.dev/python/api
Changeloghttps://crawlee.dev/python/docs/changelog
Discordhttps://discord.com/invite/jyEM2PRvMU
Stack Overflowhttps://stackoverflow.com/questions/tagged/crawlee-python
Twitterhttps://twitter.com/apify
YouTubehttps://www.youtube.com/apify
Apify platformhttps://apify.com
Docusaurushttps://docusaurus.io
GitHubhttps://github.com/apify/crawlee-python

Viewport: width=device-width, initial-scale=1.0


URLs of crawlers that visited me.