jp/www.byjp.me
1
0
Fork 0
mirror of https://github.com/by-jp/www.byjp.me.git synced 2025-08-10 06:35:41 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Remove a bunch of AI crawlers

Browse source 
Pulled from Cory's post: https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers/
This commit is contained in:
JP Hastings-Spital 2024-03-27 21:09:17 +00:00
parent 2388d1f55b
commit 05b5f19b26
1 changed files with 26 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
layouts/robots.txt
View file

@ -1,4 +1,30 @@
Sitemap: {{"/sitemap.xml" | absURL}}
User-agent: *
Disallow:
User-agent: AdsBot-Google
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: Applebot
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: DataForSeoBot
User-agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ImagesiftBot
User-agent: magpie-crawler
User-agent: omgili
User-agent: omgilibot
User-agent: peer39_crawler
User-agent: peer39_crawler/1.0
User-agent: PerplexityBot
User-agent: YouBot
Disallow: /