Title: Perplexity Crawlers - Perplexity
Open Graph Title: Perplexity Crawlers - Perplexity
X Title: Perplexity Crawlers - Perplexity
Description: We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers ("robots") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.
Open Graph Description: We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers ("robots") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.
X Description: We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers ("robots") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.
Opengraph URL: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Generator: Mintlify
Domain: perplexity.ai
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://docs.perplexity.ai/#organization","name":"Perplexity","url":"https://docs.perplexity.ai","logo":{"@type":"ImageObject","url":"https://mintcdn.com/perplexity/gHf_T6roVjp3yaJv/logo/Perplexity_API_Platform.svg?fit=max&auto=format&n=gHf_T6roVjp3yaJv&q=85&s=07cab879ca2ce031bbc3da8b20f7ab90"}},{"@type":"WebSite","@id":"https://docs.perplexity.ai#website","name":"Perplexity","url":"https://docs.perplexity.ai","publisher":{"@id":"https://docs.perplexity.ai/#organization"}},{"@type":"WebPage","@id":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers#webpage","url":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers","name":"Perplexity Crawlers","description":"We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers (\"robots\") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.","dateModified":"2026-10-29T12:00:15.200Z","isPartOf":{"@id":"https://docs.perplexity.ai#website"},"breadcrumb":{"@id":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers#breadcrumb"}},{"@type":"BreadcrumbList","@id":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Resources","item":"https://docs.perplexity.ai/docs/getting-started/integrations/ag2"},{"@type":"ListItem","position":2,"name":"Perplexity Crawlers","item":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers"}]},{"@type":["Article","TechArticle"],"@id":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers#article","headline":"Perplexity Crawlers","name":"Perplexity Crawlers","description":"We strive to improve our service every day by delivering the best search experience possible. To achieve this, we collect data using web crawlers (\"robots\") and user agents that gather and index information from the internet, operating either automatically or in response to user requests. Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity. Each setting works independently, and it may take up to 24 hours for our systems to reflect changes.","url":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers","mainEntityOfPage":{"@id":"https://docs.perplexity.ai/docs/resources/perplexity-crawlers#webpage"},"dateModified":"2026-10-29T12:00:15.200Z","publisher":{"@id":"https://docs.perplexity.ai/#organization"},"isPartOf":{"@id":"https://docs.perplexity.ai#website"}}]}| application-name | Perplexity |
| og:site_name | Perplexity |
| og:type | website |
| og:image | https://perplexity.mintlify.app/_mintlify/api/og/v2?division=Resources&title=Perplexity+Crawlers&description=We+strive+to+improve+our+service+every+day+by+delivering+the+best+search+experience+possible.+To+achieve+this,+we+collect+data+using+web+crawlers+("robots")+and&theme=0c682f4b9a1b4bccaac4b8e9&signature=382e751eec65c490e139b4ea04cff4a872770036d33a3815efe7f218b3431df8 |
| og:image:width | 1200 |
| og:image:height | 630 |
| twitter:card | summary_large_image |
| twitter:image | https://perplexity.mintlify.app/_mintlify/api/og/v2?division=Resources&title=Perplexity+Crawlers&description=We+strive+to+improve+our+service+every+day+by+delivering+the+best+search+experience+possible.+To+achieve+this,+we+collect+data+using+web+crawlers+("robots")+and&theme=0c682f4b9a1b4bccaac4b8e9&signature=382e751eec65c490e139b4ea04cff4a872770036d33a3815efe7f218b3431df8 |
| twitter:image:width | 1200 |
| twitter:image:height | 630 |
| msapplication-config | /mintlify-assets/_mintlify/favicons/perplexity/rSPP83rcZL_iw-xo/_generated/favicon/browserconfig.xml |
| apple-mobile-web-app-title | Perplexity |
| msapplication-TileColor | #121516 |
| google-site-verification | l-0uTKc1Uf60LaNt0Pkw1wLOBFEKoBIhUrv0EMr1J4U |
Links:
Viewport: width=device-width, initial-scale=1, viewport-fit=cover