René's URL Experiment: Known Crawlers

• https://about.censys.io
• https://anthropic.com/bot
• https://ai.meta.com
• https://ai.google
• https://api.slack.com/robots
• http://www.apple.com/go/applebot
• https://awario.com/bots.html
• https://babbar.tech/crawler
• http://www.baidu.com/search/spider.html
• ClaudeBot@anthropic.com
• https://dataforseo.com/dataforseo-bot
• https://developer.amazon.com/support/amazonbot
• https://developers.facebook.com/docs/sharing/webmasters/crawler
• https://discordapp.com
• http://duckduckgo.com/duckassistbot.html
• https://gemini.google
• http://help.coccoc.com/searchengine
• https://help.qwant.com/bot/
• https://internet-measurement.com/
• https://iask.ai/
• https://ibou.io/iboubot.html
• https://leakix.net
• https://modat.io
• https://www.mojeek.com/bot.html
• https://github.com/adbar/trafilatura
• https://openai.com/gptbot
• https://openai.com/searchbot
• https://opensiteexplorer.org/dotbot
• https://perplexity.ai/perplexity-user
• https://perplexity.ai/perplexitybot
• https://www.plagaware.com/bot
• https://scrapy.org
• http://www.sogou.com/docs/help/webmasters.htm#07
• http://sur.ly/bot.html
• https://help.seranking.com/en/blex-crawler
• https://support.google.com/webmasters/answer/1061943
• https://webmaster.petalsearch.com/site/petalbot
• http://yandex.com/bots
• http://www.apple.com/go/applebot
• http://www.bing.com/bingbot.htm
• http://www.facebook.com/externalhit_uatext.php
• https://www.letsencrypt.org
• http://www.linkedin.com
• http://www.google.com/bot.html
• http://www.google.com/mobile/adsbot.html
• https://www.nokia.com/genomecrawler
• http://www.zombiedomain.net/robot/
• https://x.ai