cblgh
|
9377bd6fab
|
go fmt
|
2022-11-22 14:08:59 +01:00 |
cblgh
|
9517f62de2
|
tweak wording and minor details relating to preview queries
|
2022-11-22 14:08:44 +01:00 |
Slatian
|
ed5f5189b0
|
Added a little check for the response code to not index pages that return errors or finish with codes in the 100 range
|
2022-11-22 13:52:32 +01:00 |
Slatian
|
5fe32df938
|
Added some extra mechanisms that come in handy for getting more useful previews
|
2022-11-22 13:52:32 +01:00 |
Slatian
|
0a85f38b36
|
Made scring the preview text configurable and improved the cleanup function a tiny bit.
|
2022-11-22 13:52:32 +01:00 |
cblgh
|
b0ad7dce10
|
add custom webring selector for precrawl
|
2022-03-30 15:18:38 +02:00 |
idk
|
21ef8aac08
|
Allows the configuration of a proxy (#9)
* Add proxy support, capability to crawl using SOCKS proxies
|
2022-03-29 14:36:48 +02:00 |
cblgh
|
9f912b8323
|
improve crawling rules wrt path-suffixed sites, close #6
|
2022-03-07 11:24:20 +01:00 |
cblgh
|
b9387d3b04
|
add user agent Lieu, misc fixes for null results
|
2021-12-01 09:56:09 +01:00 |
cblgh
|
44914e0f21
|
merge eotl changes :)
|
2021-05-11 20:39:14 +02:00 |
cblgh
|
28d052f4c3
|
launch
|
2021-02-03 09:12:30 +01:00 |