3 Commity (6a11d197818b1ef152f4f9f1fddb4670a0ab5886)

Autor	SHA1	Wiadomość	Data
Štěpán Škorpil	0723c2508d	Fixed robots.txt fetching	2022-11-23 23:34:05 +01:00
Štěpán Škorpil	8f42791ee2	Added timeout to robots.txt fetching	2022-11-22 18:52:24 +01:00
Štěpán Škorpil	704c7c066e	Added robots.txt handling	2022-11-22 16:37:11 +01:00