René's URL Explorer Experiment


Title: Perplexity Crawlers - Perplexity

Open Graph Title: Perplexity Crawlers - Perplexity

X Title: Perplexity Crawlers - Perplexity

Description: We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers ("robots") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.

Open Graph Description: We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers ("robots") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.

X Description: We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers ("robots") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.

Opengraph URL: https://docs.perplexity.ai/docs/resources/perplexity-crawlers

Generator: Mintlify

direct link

Domain: perplexity.ai


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"WebSite","name":"Perplexity","creator":{"@type":"Organization","name":"Mintlify","url":"https://mintlify.com"}}

application-namePerplexity
msapplication-config/mintlify-assets/_mintlify/favicons/perplexity/rSPP83rcZL_iw-xo/_generated/favicon/browserconfig.xml
apple-mobile-web-app-titlePerplexity
msapplication-TileColor#121516
google-site-verificationl-0uTKc1Uf60LaNt0Pkw1wLOBFEKoBIhUrv0EMr1J4U
og:site_namePerplexity
og:imagehttps://perplexity.mintlify.app/mintlify-assets/_next/image?url=/_mintlify/api/og?division=Resources&title=Perplexity+Crawlers&description=We+strive+to+improve+our+service+every+day+by+delivering+the+best+search+experience+possible.+To+achieve+this%2C+we+collect+data+using+web+crawlers+%28%22robots%22%29+and&logoLight=https%3A%2F%2Fmintcdn.com%2Fperplexity%2FgHf_T6roVjp3yaJv%2Flogo%2FPerplexity_API_Platform.svg%3Ffit%3Dmax%26auto%3Dformat%26n%3DgHf_T6roVjp3yaJv%26q%3D85%26s%3D07cab879ca2ce031bbc3da8b20f7ab90&logoDark=https%3A%2F%2Fmintcdn.com%2Fperplexity%2FgHf_T6roVjp3yaJv%2Flogo%2FPerplexity_API_Platform_Light.svg%3Ffit%3Dmax%26auto%3Dformat%26n%3DgHf_T6roVjp3yaJv%26q%3D85%26s%3D416d5342d11094201b851dcf3eabdc9d&primaryColor=%23121516&lightColor=%23F7F7F8&darkColor=%23121516&backgroundLight=%23F7F7F8&backgroundDark=%23121516&w=1200&q=100
og:image:width1200
og:image:height630
og:typewebsite
twitter:cardsummary_large_image
twitter:imagehttps://perplexity.mintlify.app/mintlify-assets/_next/image?url=/_mintlify/api/og?division=Resources&title=Perplexity+Crawlers&description=We+strive+to+improve+our+service+every+day+by+delivering+the+best+search+experience+possible.+To+achieve+this%2C+we+collect+data+using+web+crawlers+%28%22robots%22%29+and&logoLight=https%3A%2F%2Fmintcdn.com%2Fperplexity%2FgHf_T6roVjp3yaJv%2Flogo%2FPerplexity_API_Platform.svg%3Ffit%3Dmax%26auto%3Dformat%26n%3DgHf_T6roVjp3yaJv%26q%3D85%26s%3D07cab879ca2ce031bbc3da8b20f7ab90&logoDark=https%3A%2F%2Fmintcdn.com%2Fperplexity%2FgHf_T6roVjp3yaJv%2Flogo%2FPerplexity_API_Platform_Light.svg%3Ffit%3Dmax%26auto%3Dformat%26n%3DgHf_T6roVjp3yaJv%26q%3D85%26s%3D416d5342d11094201b851dcf3eabdc9d&primaryColor=%23121516&lightColor=%23F7F7F8&darkColor=%23121516&backgroundLight=%23F7F7F8&backgroundDark=%23121516&w=1200&q=100
twitter:image:width1200
twitter:image:height630

Links:

/llms.txthttps://perplexity.ai/llms.txt
Skip to main contenthttps://perplexity.ai/perplexity-user#content-area
llms.txthttps://perplexity.ai/llms.txt
Perplexity home pagehttps://perplexity.ai/docs/getting-started/overview
Docshttps://perplexity.ai/docs/getting-started/overview
Cookbookhttps://perplexity.ai/docs/cookbook
API Referencehttps://perplexity.ai/api-reference/agent-post
Get Startedhttps://console.perplexity.ai
Get Startedhttps://console.perplexity.ai
Communityhttps://community.perplexity.ai
Bloghttps://research.perplexity.ai/articles
Changeloghttps://perplexity.ai/docs/resources/changelog
Overviewhttps://perplexity.ai/docs/getting-started/overview
Quickstarthttps://perplexity.ai/docs/getting-started/quickstart
Pricinghttps://perplexity.ai/docs/getting-started/pricing
Perplexity MCP Serverhttps://perplexity.ai/docs/getting-started/integrations/mcp-server
Overviewhttps://perplexity.ai/docs/agent-api/quickstart
OpenAI Compatibilityhttps://perplexity.ai/docs/agent-api/openai-compatibility
Prompt Guidehttps://perplexity.ai/docs/agent-api/prompt-guide
Quickstarthttps://perplexity.ai/docs/search/quickstart
Best Practiceshttps://perplexity.ai/docs/search/best-practices
Quickstarthttps://perplexity.ai/docs/sonar/quickstart
Prompt Guidehttps://perplexity.ai/docs/sonar/prompt-guide
OpenAI SDK Compatibilityhttps://perplexity.ai/docs/sonar/openai-compatibility
Quickstarthttps://perplexity.ai/docs/embeddings/quickstart
Best Practiceshttps://perplexity.ai/docs/embeddings/best-practices
Standard Embeddingshttps://perplexity.ai/docs/embeddings/standard-embeddings
Contextualized Embeddingshttps://perplexity.ai/docs/embeddings/contextualized-embeddings
Quickstarthttps://perplexity.ai/docs/sdk/overview
API Groups & Billinghttps://perplexity.ai/docs/getting-started/api-groups
API Key Managementhttps://perplexity.ai/docs/admin/api-key-management
Computer Analytics APIhttps://perplexity.ai/docs/admin/computer-analytics-api
Rate Limits & Usage Tiershttps://perplexity.ai/docs/admin/rate-limits-usage-tiers
AWS Marketplacehttps://perplexity.ai/docs/resources/aws-marketplace
API Roadmaphttps://perplexity.ai/docs/resources/feature-roadmap
Privacy & Securityhttps://perplexity.ai/docs/resources/privacy-security
Frequently Asked Questionshttps://perplexity.ai/docs/resources/faq
System Statushttps://perplexity.ai/docs/resources/status
Changeloghttps://perplexity.ai/docs/resources/changelog
Get in Touchhttps://perplexity.ai/docs/resources/discussions
Perplexity Crawlershttps://perplexity.ai/docs/resources/perplexity-crawlers
WAF Configurationhttps://perplexity.ai/perplexity-user#waf-configuration
Cloudflare WAFhttps://perplexity.ai/perplexity-user#cloudflare-waf
AWS WAFhttps://perplexity.ai/perplexity-user#aws-waf
IP Address Sourceshttps://perplexity.ai/perplexity-user#ip-address-sources
Best Practiceshttps://perplexity.ai/perplexity-user#best-practices
https://www.perplexity.com/perplexitybot.jsonhttps://www.perplexity.com/perplexitybot.json
https://www.perplexity.com/perplexity-user.jsonhttps://www.perplexity.com/perplexity-user.json
https://perplexity.ai/perplexity-user#waf-configuration
https://perplexity.ai/perplexity-user#cloudflare-waf
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user#aws-waf
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user
https://perplexity.ai/perplexity-user#ip-address-sources
https://www.perplexity.com/perplexitybot.jsonhttps://www.perplexity.com/perplexitybot.json
https://www.perplexity.com/perplexity-user.jsonhttps://www.perplexity.com/perplexity-user.json
https://perplexity.ai/perplexity-user#best-practices
Get in TouchPrevioushttps://perplexity.ai/docs/resources/discussions
xhttps://x.com/PPLXDevs
discordhttps://discord.com/invite/perplexity-ai
websitehttps://community.perplexity.ai
Powered byThis documentation is built and hosted on Mintlify, a developer documentation platformhttps://www.mintlify.com?utm_campaign=poweredBy&utm_medium=referral&utm_source=perplexity

Viewport: width=device-width, initial-scale=1, viewport-fit=cover


URLs of crawlers that visited me.