mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
791 Commity
15 Gałęzie
109 Tagi
19 MiB
Wykres commitów

791 Commity (fade68c6f48bcc6cc69c6dcf05e4b398e5439dd0)

Autor SHA1 Wiadomość Data
Patrick Robertson fade68c6f4 Fix up unit tests - dataclass + subclasses not having @dataclass was breaking it 2025-01-30 13:45:24 +01:00
Patrick Robertson b7d9145f6c Further tidyups + refactoring for new structure 
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
 2025-01-30 13:21:10 +01:00
erinhmclark cddae65a90 Update modules for new core structure. 2025-01-30 08:42:23 +00:00
Patrick Robertson 18ff36ce15 Add ruamel to dependencies (replaces pyyaml) 2025-01-29 19:37:41 +01:00
Patrick Robertson 00a7018f36 Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember 2025-01-29 19:25:22 +01:00
Patrick Robertson 3d37c494aa Tidy ups + unit tests: 
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
 2025-01-29 18:42:49 +01:00
Patrick Robertson dcd5576f29 set metadata enricher to requires_setup=True (requires exiftool which isn't installed by default on most machines) 2025-01-29 00:10:40 +01:00
Patrick Robertson 7a4871db6b Fix up unit tests for new structure 2025-01-28 14:40:12 +01:00
Patrick Robertson 9635449ac0 more user friendly error logging when config issues are found 2025-01-28 11:44:52 +01:00
Patrick Robertson 27b25c5bd4 Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them 2025-01-28 11:37:23 +01:00
Patrick Robertson 1d2a1d4db7 Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls 
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
 2025-01-28 11:14:12 +01:00
erinhmclark 57b3bec935 Google sheets feeder and database implemented. 2025-01-27 20:13:12 +00:00
erinhmclark 6c67effd8c remove name reference in local_storage.py 2025-01-27 19:17:18 +00:00
erinhmclark e1a9373336 Refactoring for new config setup 2025-01-27 19:03:02 +00:00
Patrick Robertson e3074013d0 Fix loading/saving to orchestration file with comments 2025-01-27 14:28:04 +01:00
Patrick Robertson f68e2726f2 Refactor loader + step into module, use LazyBaseModule and BaseModule 2025-01-27 14:01:36 +01:00
Patrick Robertson 7fd95866a1 Further fixes/changes to loading 'types' for config + manifest edits 2025-01-27 11:48:04 +01:00
Patrick Robertson 14e2479599 Merge branch 'more_mainifests' into load_modules 2025-01-27 11:05:56 +01:00
Patrick Robertson 0b03f54f4e Fix up config validation, and allow for custom 'validators' 2025-01-27 11:00:52 +01:00
erinhmclark ebebd27897 Fix archiver to extractor naming 2025-01-27 09:11:45 +00:00
erinhmclark 21a7ff0520 Fix types in manifests 2025-01-27 08:43:18 +00:00
erinhmclark 96b35a272c Rm gsheet references in utils 2025-01-24 18:51:15 +00:00
erinhmclark dd402b456f Fix and add types to manifest 2025-01-24 18:50:11 +00:00
Patrick Robertson 3fc6ddfe85 Tweaks to logging strings 2025-01-24 15:30:00 +01:00
Patrick Robertson f1e9ab6751 Merge branch 'main' into load_modules 2025-01-24 15:23:15 +01:00
Patrick Robertson e8138eac1c
Add ubuntu-latest to the matrix of test runners (#181) 
* Don't clutter logs with info about generic dropin

* Add ubuntu-latest to unit tests

This is currently failing due to an issue with oscrypto and newer openssl https://github.com/wbond/oscrypto/issues/78#issuecomment-1756317472

* fix oscrypto version for ubuntu 24 compatibility (boto3 too see #180)

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
 2025-01-24 14:03:55 +00:00
Miguel Sozinho Ramalho a6fc4e1bb1
modifies base docker image to use browsertrix 1.4.2 (#182) 
* modifies base image to newest browsertrix version

* modify browsertrix cmd args based on recent experience
 2025-01-24 13:59:29 +00:00
erinhmclark 1942e8b819 Gsheets utility revert 2025-01-24 13:34:30 +00:00
erinhmclark 024fe58377 fix config parsing in manifests, remove module level configs 2025-01-24 13:33:12 +00:00
erinhmclark 0453d95f56 fix config parsing in manifests 2025-01-24 13:24:54 +00:00
erinhmclark aa7ca93a43 Update manifests and modules 2025-01-24 12:58:16 +00:00
erinhmclark ba4b330881 Merge remote-tracking branch 'origin/more_mainifests' into more_mainifests 2025-01-24 08:04:27 +00:00
erinhmclark cbafbfab3f Revert Dockerfile changes 2025-01-24 08:04:09 +00:00
Patrick Robertson 9befb9776c Fix loading modules when entry_point isn't set 2025-01-23 21:08:54 +01:00
Patrick Robertson 06f6e34d9d Revert changes to orchestrator to avoid merge conflicts 2025-01-23 20:38:36 +01:00
Patrick Robertson b27bf8ffeb Fix up loading/storing configs + unit tests 2025-01-23 20:32:19 +01:00
erinhmclark 50f4ebcdc3 Move storage configs into individual manifests, assert format on useage. 2025-01-23 17:01:30 +00:00
erinhmclark c3403ced26 Rename storages for clarity 2025-01-23 16:51:17 +00:00
erinhmclark 1274a1b231 More manifests, base modules and rename from archiver to extractor. 2025-01-23 16:40:48 +00:00
erinhmclark 9db26cdfc2 Merge branch 'load_modules' into more_mainifests 
# Conflicts:
#	src/auto_archiver/core/orchestrator.py
 2025-01-23 09:19:54 +00:00
erinhmclark 79684f8348 Set up feeder manifests (not merged by source yet) 2025-01-23 09:16:42 +00:00
Patrick Robertson 65ef46d01e Fix loading already loaded modules - don't load them twice 2025-01-23 00:09:39 +01:00
Patrick Robertson 550097ab7b Get module loading working properly 2025-01-22 23:54:21 +01:00
erinhmclark c517d35bdf Merge branch 'load_modules' into more_mainifests 
# Conflicts:
#	src/auto_archiver/databases/__init__.py
 2025-01-22 18:19:43 +00:00
erinhmclark 99c8c69085 Manifests for databases 2025-01-22 18:18:13 +00:00
Patrick Robertson ade5ea0f6f Tidy up imports + start on loading modules - program now starts much faster 2025-01-22 18:45:58 +01:00
Patrick Robertson b6b085854c Switch back to using yaml with dot notation 
(two simple helper functions to convert between dot and dict notation)
 2025-01-22 17:40:51 +01:00
Patrick Robertson 54995ad6ab Further tweaks based on __manifest__.py files 
Loading configs now works
 2025-01-22 13:11:43 +01:00
erinhmclark 7b3a1468cd Create manifest files for archiver modules. 2025-01-22 10:21:27 +01:00
Patrick Robertson 4830f99300 Get parsing of manifest and combining with config file working 2025-01-21 20:03:10 +01:00