11 Commity (main)

Autor	SHA1	Wiadomość	Data
cblgh	9377bd6fab	go fmt	2022-11-22 14:08:59 +01:00
cblgh	9517f62de2	tweak wording and minor details relating to preview queries	2022-11-22 14:08:44 +01:00
Slatian	ed5f5189b0	Added a little check for the response code to not index pages that return errors or finish with codes in the 100 range	2022-11-22 13:52:32 +01:00
Slatian	5fe32df938	Added some extra mechanisms that come in handy for getting more useful previews	2022-11-22 13:52:32 +01:00
Slatian	0a85f38b36	Made scring the preview text configurable and improved the cleanup function a tiny bit.	2022-11-22 13:52:32 +01:00
cblgh	b0ad7dce10	add custom webring selector for precrawl	2022-03-30 15:18:38 +02:00
idk	21ef8aac08	Allows the configuration of a proxy (#9) ... * Add proxy support, capability to crawl using SOCKS proxies	2022-03-29 14:36:48 +02:00
cblgh	9f912b8323	improve crawling rules wrt path-suffixed sites, close #6	2022-03-07 11:24:20 +01:00
cblgh	b9387d3b04	add user agent Lieu, misc fixes for null results	2021-12-01 09:56:09 +01:00
cblgh	44914e0f21	merge eotl changes :)	2021-05-11 20:39:14 +02:00
cblgh	28d052f4c3	launch	2021-02-03 09:12:30 +01:00