mirror
/
fedicrawl
kopia lustrzana https://github.com/Stopka/fedicrawl
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
38 Commity
1 Gałąź
0 Tagi
407 KiB
Wykres commitów

35 Commity (main)

Autor SHA1 Wiadomość Data
Štěpán Škorpil 4fbfce7f12 Added support for popular Pleroma and Misskey forks 2023-01-07 21:03:17 +01:00
Štěpán Škorpil 87f5ddc8b1 If nodeinfo refresh is failed, software name and version is unset 2022-12-27 22:40:15 +01:00
Štěpán Škorpil a8e6dbdb9d Fixed old feed deleting 2022-12-27 22:11:19 +01:00
Štěpán Škorpil 425abd5af0 Added domain validation 2022-12-10 15:11:56 +01:00
Štěpán Škorpil 1cbd5df5ae Added config for seed domain timeout 2022-11-29 14:11:27 +01:00
Štěpán Škorpil 6a11d19781 Added crawling depth limit 2022-11-24 00:39:17 +01:00
Štěpán Škorpil 0723c2508d Fixed robots.txt fetching 2022-11-23 23:34:05 +01:00
Štěpán Škorpil 4064db521f Added promise batching 2022-11-23 23:19:28 +01:00
Štěpán Škorpil 8f42791ee2 Added timeout to robots.txt fetching 2022-11-22 18:52:24 +01:00
Štěpán Škorpil 59f49ce29d Changed tag from noindex to nobot 2022-11-22 18:39:05 +01:00
Štěpán Škorpil c0b10f2e7a It respects noindex option in mastodon 2022-11-22 17:35:34 +01:00
Štěpán Škorpil 4113d78a17 It respects noindex option in mastodon 2022-11-22 17:01:25 +01:00
Štěpán Škorpil 704c7c066e Added robots.txt handling 2022-11-22 16:37:11 +01:00
Štěpán Škorpil 8d0452d16b Set mastodon page limit to 40 2022-11-22 11:00:24 +01:00
Štěpán Škorpil 45cafbe94b Set mastodon page limit to 40 2022-11-22 10:57:34 +01:00
Štěpán Škorpil 29acce3906 Properly applied code quality tools 2022-09-18 13:32:25 +02:00
Štěpán Škorpil 4415c27b1b Deploy fixes 2022-09-14 21:47:11 +02:00
Štěpán Škorpil f6f9d758a1 Replaced postgresql by elastic search 2022-09-14 21:16:04 +02:00
Štěpán Škorpil 7eb28e5d84 Enabled support for friendica 2022-05-29 15:55:53 +02:00
Štěpán Škorpil b2725e5acf Added nodeinfo content length limit (5k) solving issues with global redirects to endless streams 2022-04-18 22:18:43 +02:00
Štěpán Škorpil cc42eebe0e Added nodeinfo content length limit (5k) solving issues with global redirects to endless streams 2022-04-18 22:09:24 +02:00
Štěpán Škorpil 3ed37c04ce Not existing feeds are deleted from databse 2022-04-17 14:03:53 +02:00
Štěpán Škorpil cefbd76066 Added support for ecko and hometown 2022-03-04 20:17:53 +01:00
Štěpán Škorpil 5c0a032ed7 Software name forced to lower case 2022-01-15 01:48:45 +01:00
Štěpán Škorpil b2eab401ec Fixed Misskey nullable user properties 2022-01-08 19:25:44 +01:00
Štěpán Škorpil 32ad01a849 Added misskey 2022-01-08 19:13:56 +01:00
Štěpán Škorpil 945c9495f5 Added pleroma 2022-01-08 02:16:16 +01:00
Štěpán Škorpil de5e989bbb Fixed refactoring mess 2022-01-07 23:56:40 +01:00
Štěpán Škorpil 45bb0569c0 Improved peertube description parsing 2022-01-03 14:05:00 +01:00
Štěpán Škorpil 0c2ade0b52 Added more env configs 2022-01-03 13:26:17 +01:00
Štěpán Škorpil 8330ed53c4 Fixed README.md 2022-01-01 17:33:03 +01:00
Štěpán Škorpil b5df2b986a Added addresses to fulltext 2022-01-01 12:21:43 +01:00
Štěpán Škorpil d2ab8bfb2f Add display name to fulltext field, fixed waiting when all feeds are processed 2021-12-28 00:20:19 +01:00
Štěpán Škorpil 3acd0ec72f Added fulltext feed column 2021-12-23 18:46:43 +01:00
Štěpán Škorpil 74e7dca8bd Initial crawler commit 2021-12-23 15:14:06 +01:00