1
0
Fork 1
mirror of https://example.com synced 2024-11-27 06:56:40 +09:00
firefish/custom/assets/robots.txt
ThatOneCalculator 86cad1056c
docs: 🔒 robots.txt asset
Closes #10640, #10608
2023-09-02 11:44:31 -07:00

14 lines
236 B
Text

User-agent: *
Allow: /
# Uncomment the following to block CommonCrawl
#
# User-agent: CCBot
# User-agent: CCBot/2.0
# User-agent: CCBot/3.1
# Disallow: /
# Uncomment the following to block ChatGPT
#
# User-agent: GPTBot
# Disallow: /