mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 033 Commity
9 Gałęzie
112 Tagi
22 MiB
Wykres commitów

1033 Commity (7e10040bbd7f51efa9359924f247c57aa502e89c)

Autor SHA1 Wiadomość Data
Patrick Robertson 9635449ac0 more user friendly error logging when config issues are found 2025-01-28 11:44:52 +01:00
Patrick Robertson 27b25c5bd4 Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them 2025-01-28 11:37:23 +01:00
Patrick Robertson 1d2a1d4db7 Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls 
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
 2025-01-28 11:14:12 +01:00
erinhmclark 57b3bec935 Google sheets feeder and database implemented. 2025-01-27 20:13:12 +00:00
erinhmclark 6c67effd8c remove name reference in local_storage.py 2025-01-27 19:17:18 +00:00
erinhmclark e1a9373336 Refactoring for new config setup 2025-01-27 19:03:02 +00:00
Patrick Robertson e3074013d0 Fix loading/saving to orchestration file with comments 2025-01-27 14:28:04 +01:00
Patrick Robertson f68e2726f2 Refactor loader + step into module, use LazyBaseModule and BaseModule 2025-01-27 14:01:36 +01:00
Patrick Robertson 7fd95866a1 Further fixes/changes to loading 'types' for config + manifest edits 2025-01-27 11:48:04 +01:00
Patrick Robertson 14e2479599 Merge branch 'more_mainifests' into load_modules 2025-01-27 11:05:56 +01:00
Patrick Robertson 0b03f54f4e Fix up config validation, and allow for custom 'validators' 2025-01-27 11:00:52 +01:00
erinhmclark ebebd27897 Fix archiver to extractor naming 2025-01-27 09:11:45 +00:00
erinhmclark 21a7ff0520 Fix types in manifests 2025-01-27 08:43:18 +00:00
erinhmclark 96b35a272c Rm gsheet references in utils 2025-01-24 18:51:15 +00:00
erinhmclark dd402b456f Fix and add types to manifest 2025-01-24 18:50:11 +00:00
Patrick Robertson 3fc6ddfe85 Tweaks to logging strings 2025-01-24 15:30:00 +01:00
Patrick Robertson f1e9ab6751 Merge branch 'main' into load_modules 2025-01-24 15:23:15 +01:00
Patrick Robertson e8138eac1c
Add ubuntu-latest to the matrix of test runners (#181) 
* Don't clutter logs with info about generic dropin

* Add ubuntu-latest to unit tests

This is currently failing due to an issue with oscrypto and newer openssl https://github.com/wbond/oscrypto/issues/78#issuecomment-1756317472

* fix oscrypto version for ubuntu 24 compatibility (boto3 too see #180)

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
 2025-01-24 14:03:55 +00:00
Miguel Sozinho Ramalho a6fc4e1bb1
modifies base docker image to use browsertrix 1.4.2 (#182) 
* modifies base image to newest browsertrix version

* modify browsertrix cmd args based on recent experience
 2025-01-24 13:59:29 +00:00
erinhmclark 1942e8b819 Gsheets utility revert 2025-01-24 13:34:30 +00:00
erinhmclark 024fe58377 fix config parsing in manifests, remove module level configs 2025-01-24 13:33:12 +00:00
erinhmclark 0453d95f56 fix config parsing in manifests 2025-01-24 13:24:54 +00:00
erinhmclark aa7ca93a43 Update manifests and modules 2025-01-24 12:58:16 +00:00
erinhmclark ba4b330881 Merge remote-tracking branch 'origin/more_mainifests' into more_mainifests 2025-01-24 08:04:27 +00:00
erinhmclark cbafbfab3f Revert Dockerfile changes 2025-01-24 08:04:09 +00:00
Patrick Robertson 9befb9776c Fix loading modules when entry_point isn't set 2025-01-23 21:08:54 +01:00
Patrick Robertson 06f6e34d9d Revert changes to orchestrator to avoid merge conflicts 2025-01-23 20:38:36 +01:00
Patrick Robertson b27bf8ffeb Fix up loading/storing configs + unit tests 2025-01-23 20:32:19 +01:00
erinhmclark 50f4ebcdc3 Move storage configs into individual manifests, assert format on useage. 2025-01-23 17:01:30 +00:00
erinhmclark c3403ced26 Rename storages for clarity 2025-01-23 16:51:17 +00:00
erinhmclark 1274a1b231 More manifests, base modules and rename from archiver to extractor. 2025-01-23 16:40:48 +00:00
erinhmclark 9db26cdfc2 Merge branch 'load_modules' into more_mainifests 
# Conflicts:
#	src/auto_archiver/core/orchestrator.py
 2025-01-23 09:19:54 +00:00
erinhmclark 79684f8348 Set up feeder manifests (not merged by source yet) 2025-01-23 09:16:42 +00:00
Patrick Robertson 65ef46d01e Fix loading already loaded modules - don't load them twice 2025-01-23 00:09:39 +01:00
Patrick Robertson 550097ab7b Get module loading working properly 2025-01-22 23:54:21 +01:00
erinhmclark c517d35bdf Merge branch 'load_modules' into more_mainifests 
# Conflicts:
#	src/auto_archiver/databases/__init__.py
 2025-01-22 18:19:43 +00:00
erinhmclark 99c8c69085 Manifests for databases 2025-01-22 18:18:13 +00:00
Patrick Robertson ade5ea0f6f Tidy up imports + start on loading modules - program now starts much faster 2025-01-22 18:45:58 +01:00
Patrick Robertson b6b085854c Switch back to using yaml with dot notation 
(two simple helper functions to convert between dot and dict notation)
 2025-01-22 17:40:51 +01:00
Patrick Robertson 54995ad6ab Further tweaks based on __manifest__.py files 
Loading configs now works
 2025-01-22 13:11:43 +01:00
erinhmclark 7b3a1468cd Create manifest files for archiver modules. 2025-01-22 10:21:27 +01:00
Patrick Robertson 4830f99300 Get parsing of manifest and combining with config file working 2025-01-21 20:03:10 +01:00
Patrick Robertson 241b35002c Initial changes to move to '__manifest__' format 2025-01-21 19:02:38 +01:00
Patrick Robertson 03f3770223 Add __manifest__.py for generic_extractor 2025-01-21 18:00:45 +01:00
Patrick Robertson bdfc855297 Ignore pylint statements for manifest files 2025-01-21 17:59:52 +01:00
Patrick Robertson c41d93a634 Use already implemented helper to get version 2025-01-21 17:53:37 +01:00
Patrick Robertson d4fff0b6eb
Merge pull request #175 from bellingcat/youtubedlp-rewrite 
Create generic archiver for all valid youtube-dl URLs, add truthsocial extractor, unit tests for twitter_api extractor, utility methods for cleaning HTML and traversing objects
 2025-01-21 17:33:39 +01:00
Patrick Robertson cd2ae3763f
Minor adjustments 
Co-authored-by: Miguel Sozinho Ramalho <19508417+msramalho@users.noreply.github.com>
 2025-01-21 16:24:37 +00:00
Patrick Robertson d3e3eb7639 unit tests for loading dropins 2025-01-21 16:59:45 +01:00
Patrick Robertson 9dde9b26d0 Patch in upstream changes to ytdlp for now 
Seems like ytdlp may not merge https://github.com/yt-dlp/yt-dlp/pull/12098 anytime soon
 2025-01-21 16:49:49 +01:00