mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
954 Commity
7 Gałęzie
113 Tagi
22 MiB
Wykres commitów

954 Commity (2ffe124d952dfdf37cfd92c70c1e76293f83e162)

Autor SHA1 Wiadomość Data
Patrick Robertson a873e56b87 Remove old csv_feeder file - now inside a module 2025-02-04 12:57:35 +01:00
Patrick Robertson 72b5ea9ab6 Restore headless arg 2025-02-03 17:40:40 +01:00
Patrick Robertson c574b694ed Set up screenshot enricher to use authentication/cookies 2025-02-03 17:25:59 +01:00
Patrick Robertson 7ec328ab40 Remove cookie options from generic_extractor - it now uses 'authentication' global settings :D 2025-02-03 16:04:36 +01:00
Patrick Robertson 7a2be5a0da Add cookie extraction to 'authentication' options, get generic_extractor working using this info 2025-02-03 16:03:07 +01:00
Patrick Robertson 9c9e9b370e Remove lingering reference to ArchivingContext 2025-02-03 16:02:38 +01:00
Patrick Robertson 9a8c94b641 Fix getting/setting folder context for metadata 2025-02-03 16:02:17 +01:00
Patrick Robertson c25d5cae84 Remove ArchivingContext completely 
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
 2025-01-30 17:50:54 +01:00
Patrick Robertson d76063c3f3 Fix unit tests 2025-01-30 16:46:53 +01:00
Patrick Robertson d6b4b7a932 Further cleanup 
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
 2025-01-30 16:44:40 +01:00
Patrick Robertson 953011f368 Don't make modules 'dataclasses' 2025-01-30 16:44:40 +01:00
erinhmclark 527438826c Fix manifests for required configs. 2025-01-30 13:04:51 +00:00
Patrick Robertson fade68c6f4 Fix up unit tests - dataclass + subclasses not having @dataclass was breaking it 2025-01-30 13:45:24 +01:00
Patrick Robertson b7d9145f6c Further tidyups + refactoring for new structure 
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
 2025-01-30 13:21:10 +01:00
erinhmclark cddae65a90 Update modules for new core structure. 2025-01-30 08:42:23 +00:00
Patrick Robertson 18ff36ce15 Add ruamel to dependencies (replaces pyyaml) 2025-01-29 19:37:41 +01:00
Patrick Robertson 00a7018f36 Fix up dependency checking (use 'dependencies' instead of 'external_dependencies' -> simpler/easier to remember 2025-01-29 19:25:22 +01:00
Patrick Robertson 3d37c494aa Tidy ups + unit tests: 
1. Allow loading modules from --module_paths=/extra/path/here
2. Improved unit tests for module loading
3. Further small tidy ups/clean ups
 2025-01-29 18:42:49 +01:00
Patrick Robertson 4c1c8953ca Add unit tests for timestamping_enricher 2025-01-29 12:20:52 +01:00
Patrick Robertson dcd5576f29 set metadata enricher to requires_setup=True (requires exiftool which isn't installed by default on most machines) 2025-01-29 00:10:40 +01:00
Patrick Robertson 7a4871db6b Fix up unit tests for new structure 2025-01-28 14:40:12 +01:00
Patrick Robertson 9635449ac0 more user friendly error logging when config issues are found 2025-01-28 11:44:52 +01:00
Patrick Robertson 27b25c5bd4 Validate orchestration.yaml file inputs - so if a user enters invalid values, it also validates them 2025-01-28 11:37:23 +01:00
Patrick Robertson 1d2a1d4db7 Allow framework for config settings that should not be stored in config (e.g. cli_feeder.urls 
Use 'do_not_store': True in the config settings to apply this. Also: fix up generic archiver dropins loading + local_storage defaults (same as what's in example orchestration)
 2025-01-28 11:14:12 +01:00
erinhmclark 57b3bec935 Google sheets feeder and database implemented. 2025-01-27 20:13:12 +00:00
erinhmclark 6c67effd8c remove name reference in local_storage.py 2025-01-27 19:17:18 +00:00
erinhmclark e1a9373336 Refactoring for new config setup 2025-01-27 19:03:02 +00:00
Patrick Robertson e3074013d0 Fix loading/saving to orchestration file with comments 2025-01-27 14:28:04 +01:00
Patrick Robertson f68e2726f2 Refactor loader + step into module, use LazyBaseModule and BaseModule 2025-01-27 14:01:36 +01:00
Patrick Robertson 7fd95866a1 Further fixes/changes to loading 'types' for config + manifest edits 2025-01-27 11:48:04 +01:00
Patrick Robertson 14e2479599 Merge branch 'more_mainifests' into load_modules 2025-01-27 11:05:56 +01:00
Patrick Robertson 0b03f54f4e Fix up config validation, and allow for custom 'validators' 2025-01-27 11:00:52 +01:00
erinhmclark ebebd27897 Fix archiver to extractor naming 2025-01-27 09:11:45 +00:00
erinhmclark 21a7ff0520 Fix types in manifests 2025-01-27 08:43:18 +00:00
erinhmclark 96b35a272c Rm gsheet references in utils 2025-01-24 18:51:15 +00:00
erinhmclark dd402b456f Fix and add types to manifest 2025-01-24 18:50:11 +00:00
Patrick Robertson 3fc6ddfe85 Tweaks to logging strings 2025-01-24 15:30:00 +01:00
Patrick Robertson f1e9ab6751 Merge branch 'main' into load_modules 2025-01-24 15:23:15 +01:00
Patrick Robertson e8138eac1c
Add ubuntu-latest to the matrix of test runners (#181) 
* Don't clutter logs with info about generic dropin

* Add ubuntu-latest to unit tests

This is currently failing due to an issue with oscrypto and newer openssl https://github.com/wbond/oscrypto/issues/78#issuecomment-1756317472

* fix oscrypto version for ubuntu 24 compatibility (boto3 too see #180)

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
 2025-01-24 14:03:55 +00:00
Miguel Sozinho Ramalho a6fc4e1bb1
modifies base docker image to use browsertrix 1.4.2 (#182) 
* modifies base image to newest browsertrix version

* modify browsertrix cmd args based on recent experience
 2025-01-24 13:59:29 +00:00
erinhmclark 1942e8b819 Gsheets utility revert 2025-01-24 13:34:30 +00:00
erinhmclark 024fe58377 fix config parsing in manifests, remove module level configs 2025-01-24 13:33:12 +00:00
erinhmclark 0453d95f56 fix config parsing in manifests 2025-01-24 13:24:54 +00:00
erinhmclark aa7ca93a43 Update manifests and modules 2025-01-24 12:58:16 +00:00
erinhmclark ba4b330881 Merge remote-tracking branch 'origin/more_mainifests' into more_mainifests 2025-01-24 08:04:27 +00:00
erinhmclark cbafbfab3f Revert Dockerfile changes 2025-01-24 08:04:09 +00:00
Patrick Robertson 9befb9776c Fix loading modules when entry_point isn't set 2025-01-23 21:08:54 +01:00
Patrick Robertson 06f6e34d9d Revert changes to orchestrator to avoid merge conflicts 2025-01-23 20:38:36 +01:00
Patrick Robertson b27bf8ffeb Fix up loading/storing configs + unit tests 2025-01-23 20:32:19 +01:00
erinhmclark 50f4ebcdc3 Move storage configs into individual manifests, assert format on useage. 2025-01-23 17:01:30 +00:00