takahe/static/robots.txt

kopia lustrzana https://github.com/jointakahe/takahe
User-Agent: *

# Don't allow any bot to crawl tags.
Disallow: /tags/
Disallow: /tags/*

# Don't allow bots to crawl through the proxy
Disallow: /proxy/*
A quick robots.txt to tell bots to stop crawling tags. (#321) 2022-12-30 17:14:12 +00:00			`User-Agent: *`

			`# Don't allow any bot to crawl tags.`
			`Disallow: /tags/`
			`Disallow: /tags/*`
Instruct robots to ignore proxy (#413) 2023-01-14 18:15:18 +00:00
			`# Don't allow bots to crawl through the proxy`
			`Disallow: /proxy/*`