8 Commity (4064db521fdc19c1252dc56b430021949f721440)

Autor	SHA1	Wiadomość	Data
Štěpán Škorpil	4064db521f	Added promise batching	2022-11-23 23:19:28 +01:00
Štěpán Škorpil	59f49ce29d	Changed tag from noindex to nobot	2022-11-22 18:39:05 +01:00
Štěpán Škorpil	c0b10f2e7a	It respects noindex option in mastodon	2022-11-22 17:35:34 +01:00
Štěpán Škorpil	4113d78a17	It respects noindex option in mastodon	2022-11-22 17:01:25 +01:00
Štěpán Škorpil	704c7c066e	Added robots.txt handling	2022-11-22 16:37:11 +01:00
Štěpán Škorpil	29acce3906	Properly applied code quality tools	2022-09-18 13:32:25 +02:00
Štěpán Škorpil	f6f9d758a1	Replaced postgresql by elastic search	2022-09-14 21:16:04 +02:00
Štěpán Škorpil	74e7dca8bd	Initial crawler commit	2021-12-23 15:14:06 +01:00