wagtail/docs/robots.txt

kopia lustrzana https://github.com/wagtail/wagtail
User-agent: *

# Only index `/en/stable/`
Disallow: /en/v*/
Disallow: /en/latest/
Disallow: /*/pdf/

Sitemap: https://docs.wagtail.io/sitemap.xml
try a different approach for including custom robots.txt Conflicts: docs/_html_extra/robots.txt docs/conf.py 2020-12-02 11:13:23 +00:00			`User-agent: *`

			# Only index `/en/stable/`
			`Disallow: /en/v*/`
			`Disallow: /en/latest/`
			`Disallow: /*/pdf/`

			`Sitemap: https://docs.wagtail.io/sitemap.xml`