mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 386 Commity
10 Gałęzie
110 Tagi
21 MiB
Wykres commitów

188 Commity (dev)

Autor SHA1 Wiadomość Data
erinhmclark ba9d67e4bb Merge branch 'main' into feat/yt-dlp-pots 2025-03-18 20:10:38 +00:00
erinhmclark c4e63ebd8c Add conditional check to setup bgutils token generation script. 
TODO: Update tests
 2025-03-18 14:54:57 +00:00
Patrick Robertson 89e387030d Tests for suitable URLs for tikwm 2025-03-18 10:04:03 +00:00
Patrick Robertson 59b910ec30 Merge main 2025-03-17 10:05:11 +00:00
Patrick Robertson b21467c922 Fix ruff checks 2025-03-14 12:59:37 +00:00
Patrick Robertson 17ae75fb95 Ruff fixes 2025-03-14 12:38:12 +00:00
Patrick Robertson b8da7607e8 Merge branch 'main' into opentimestamps 2025-03-14 12:36:03 +00:00
erinhmclark 72f48f0147 Fix merge conflicts. 2025-03-14 12:11:24 +00:00
erinhmclark 846474a4e2 Merge branch 'main' into linting_etc 2025-03-14 10:50:13 +00:00
Patrick Robertson b908655cc8 Remove references to litecoin + several tidy-ups 2025-03-13 17:40:00 +00:00
Patrick Robertson 2e25e59fa6 Fix unit tests - make caplog checks more robust, having added a new logger/debug call 2025-03-13 16:07:49 +00:00
Patrick Robertson 10ceb7aa15 Move tikwm extractor into a droping for the generic extractor 2025-03-13 15:59:42 +00:00
erinhmclark 0bef78b0b4 Remove autouse property of mock_sleep. 2025-03-13 15:23:35 +00:00
Patrick Robertson 15222199d9 Add unit test for if one calendar fails 2025-03-13 14:45:38 +00:00
Patrick Robertson e7489ac4c4 Tidy up opentimestamps 
* Simplify
* Don't add fake (pending) attestations if the calendar urls all have issues
* Remove unnecessary configs
* Improve docs on upgrading + verifying
 2025-03-13 14:30:33 +00:00
erinhmclark 16012df30b Revert exception check in test. 2025-03-13 13:57:04 +00:00
erinhmclark 8673bc5979 Fix unused imports and include rule. 2025-03-13 13:55:31 +00:00
erinhmclark e76551ba22 Add documentation, pre-commit hook, more make commands and 2025-03-13 13:21:32 +00:00
erinhmclark 753c3c6214 Linting tests 2025-03-12 14:27:45 +00:00
Patrick Robertson 394b8b2dd1 Improvements to opentimestamps enricher - make OTS file a sub-file of original media 2025-03-12 11:45:13 +00:00
erinhmclark 79f576be1d Run fix on tests. 2025-03-12 10:38:16 +00:00
Patrick Robertson 1423c10363 Finish off timestamping module 2025-03-12 10:24:57 +00:00
erinhmclark 8ca7698fa0 Move Makefile and fix import error with unused import. 2025-03-11 19:58:02 +00:00
Patrick Robertson 28041d94d9 Add unit tests for opentimestamps enricher 2025-03-11 17:33:54 +00:00
Patrick Robertson 89ee6f19b6 List out all valid TSAs + option for users to allow self signed if they want 2025-03-11 16:12:13 +00:00
Patrick Robertson 294033f156 Fix bug ordering tsr that only have one cert + more unit tests 2025-03-11 15:44:04 +00:00
Patrick Robertson 2ffe124d95 Add unit test for invalid digicert tsrs 2025-03-11 11:13:36 +00:00
Patrick Robertson 1db8be91db Improved unit tests for timestamping 2025-03-11 11:08:52 +00:00
erinhmclark 441f341139 Merge branch 'main' into linting_etc 
# Conflicts:
#	src/auto_archiver/core/consts.py
#	src/auto_archiver/core/orchestrator.py
#	src/auto_archiver/core/storage.py
#	src/auto_archiver/modules/local_storage/local_storage.py
#	src/auto_archiver/modules/s3_storage/s3_storage.py
#	tests/storages/test_S3_storage.py
#	tests/storages/test_local_storage.py
#	tests/storages/test_storage_base.py
 2025-03-11 10:39:47 +00:00
Patrick Robertson e2442b2f6b
Merge pull request #243 from bellingcat/fix-long-path-names 
Unit tests for storage types + fix storage too long issues for local storage
 2025-03-11 10:05:09 +00:00
Patrick Robertson 3f6acc0917 fully working timestamping enricher 2025-03-11 10:04:46 +00:00
erinhmclark 85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
Patrick Robertson 2b91dc9514 Fix up unit tests 2025-03-10 16:51:16 +00:00
Patrick Robertson 770f4c8a3d Refactoring of storage code: 
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
 2025-03-10 16:39:48 +00:00
Miguel Sozinho Ramalho 58bd38e292
Adds new extractor for tiktok via unofficial API (#237) 
* minor update to defaults in api_db

* readme typo

* adds and tests new tikwm tiktok downloader

* addresses PR comments
 2025-03-10 11:56:45 +00:00
Patrick Robertson e89a8da3b4 Unit tests for storage types + fix storage too long issues for local storage 2025-03-10 11:30:15 +00:00
Patrick Robertson be513e95aa Merge branch 'main' into merge_modules 2025-03-07 16:19:51 +00:00
erinhmclark fa1e65f54c Fix instagram_extractor.py typo, add warning to docs, and add basic regex test. 2025-03-06 16:25:38 +00:00
erinhmclark b9c2f98f46 Update Atlos tests 2025-03-05 21:24:38 +00:00
erinhmclark d775e4612e Update gsheet_feeder references in tests. 2025-03-04 14:06:46 +00:00
erinhmclark 077b56c150 Merge GSheet Feeder and Database. 2025-03-04 14:05:19 +00:00
erinhmclark a705a78632 Fix instagram_extractor.py typo in config value. 2025-03-03 21:06:09 +00:00
erinhmclark 4280791f07 Fix mocking in test_wayback_enricher.py. 2025-02-27 11:25:58 +00:00
erinhmclark 8124bb831d Merge branch 'main' into small_issues 
# Conflicts:
#	src/auto_archiver/core/base_module.py
#	src/auto_archiver/utils/misc.py
 2025-02-26 13:19:49 +00:00
erinhmclark b2e654aef9 Remove context manager from test_pdq_hash_enricher.py 2025-02-26 12:57:33 +00:00
Patrick Robertson afc117a229 Get downloading certs working 2025-02-26 09:33:56 +00:00
erinhmclark cf1219f798 Add text content into gsheet. 2025-02-25 20:06:44 +00:00
Patrick Robertson 4dcb77c29f Merge branch 'main' into timestamping_rewrite 2025-02-25 17:10:55 +00:00
erinhmclark 73b434aafc Tests for test_vk_extractor.py. 2025-02-25 14:08:28 +00:00
erinhmclark 2d276cb9c4 Fix tmp test file. 2025-02-25 14:08:14 +00:00
Patrick Robertson 898faf6fe4 Further WIP - currently working on verify_signed 2025-02-25 12:08:08 +00:00
Patrick Robertson f8e846d59a Create facebook dropin - working for images + text. CAVEAT: only gets the first ~100 chars of the post at the moment 2025-02-25 11:44:35 +00:00
Patrick Robertson 01bf88a695 Merge branch 'main' into timestamping_rewrite 2025-02-24 12:03:14 +00:00
Patrick Robertson 73a2e2d752 Fix tests for moving orchestration to secrets/orchestration.yaml 2025-02-21 19:05:39 +00:00
Patrick Robertson 4174285898 Fix unit tests 2025-02-20 13:18:06 +00:00
Patrick Robertson 40b8359348 Implementation test with 2 x orchestrators with different configs 2025-02-20 11:18:28 +00:00
Patrick Robertson 7dde8d609d Merge main 2025-02-20 10:29:57 +00:00
Patrick Robertson 5211c5de18
Merge pull request #210 from bellingcat/logger_fix 
Fix issue #200 + Refactor _LAZY_LOADED_MODULES
 2025-02-19 15:11:42 +00:00
erinhmclark 47a634fc63 Add WACZ, Wayback and local storage tests. 2025-02-19 13:14:08 +00:00
Patrick Robertson a9802dd004 Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules 2025-02-19 12:25:35 +00:00
Patrick Robertson 222a94563f WIP: Docs tidyups+add howto on logging and authentication 
(Authentication is WIP)
 2025-02-19 10:37:04 +00:00
erinhmclark 10a5ad62b8 Include Atlos tests, metadata fixture. 2025-02-19 09:18:41 +00:00
erinhmclark f0fd9bf445 Updates tests to use pytest-mock. 2025-02-18 23:32:03 +00:00
erinhmclark 657fbd357d Merge branch 'main' into tests/add_module_tests 2025-02-18 19:47:47 +00:00
erinhmclark 7b88df72cb Update test_metadata_enricher.py 2025-02-18 19:46:57 +00:00
Patrick Robertson 3c543a3a6a
Various fixes for issues with new architecture (#208) 
* Add formatters to the TOC - fixes #204

* Add 'steps' settings to the example YAML in the docs. Fixes #206

* Improved docs on authentication architecture

* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending

* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)

* Rename the command line entrypoint to _command_line_run

Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)

* Fix unit tests to use now code-entry points

* Version bump

* Move iterating of generator up to __main__

* Breakpoint

* two minor fixes

* Fix unit tests + add new '__main__' entry point implementation test

* Skip youtube tests if running on CI. Should still run them locally

* Fix full implementation run on GH actions

* Fix skipif test for GH Actions CI

* Add skipifs for truth - it blocks GH:

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
 2025-02-18 19:10:09 +00:00
erinhmclark ce5a200d1f Added tests, updated instagram_tbot_extractor.py raise failure. 2025-02-18 12:59:10 +00:00
erinhmclark f4c623b11b Merge branch 'main' into tests/add_module_tests 2025-02-17 09:03:04 +00:00
Patrick Robertson 6d43bc7d4d
Fix generator programmatic setup (#197) 
* Fix returning a generator of a generator

* Move download test test to pytest.mark.download
 2025-02-15 17:36:44 +00:00
erinhmclark 71b41dd901 Remove accidental path, yet again. 2025-02-14 10:05:32 +00:00
erinhmclark b0756a6a34 Remove accidental full path. 2025-02-14 09:57:44 +00:00
erinhmclark 319c1e8f92 Add more tests. 2025-02-14 09:48:37 +00:00
erinhmclark 3fce593aad Merge branch 'main' into tests/add_module_tests 2025-02-12 19:33:29 +00:00
erinhmclark cbe98c729d Enricher tests 2025-02-12 19:32:40 +00:00
erinhmclark d9d936c2ca Thumbnail enricher fix seconds to minutes. 2025-02-12 12:22:27 +00:00
Patrick Robertson d0c379a3ba WIP - timestamping enricher 2025-02-11 18:18:19 +00:00
Patrick Robertson 3163cb793a Fix timestamping enricher for new module structure (temp paths) 2025-02-11 15:26:40 +00:00
Patrick Robertson 7bb4d68a22 Merge branch 'load_modules' into timestamping_rewrite 2025-02-11 15:21:31 +00:00
Patrick Robertson 29901da601 Merge branch 'load_modules' into docs_update 2025-02-11 14:10:56 +00:00
Patrick Robertson 2f51d3917a Further addition to docs: creating modules, configurations, installation 2025-02-11 13:49:30 +00:00
erinhmclark d1d6cde008 Set mock timestamp without z format 2025-02-11 12:27:48 +00:00
erinhmclark 5e2e93382f Test fixes for 3.10 compliance. 2025-02-11 12:17:42 +00:00
erinhmclark f97ec6a9e0 Fixed S3 module import 2025-02-11 11:58:28 +00:00
erinhmclark 89d9140d15 Fixed setup/ config_setup reference 2025-02-11 11:47:11 +00:00
erinhmclark 1792e02d1d skip authenticated tests in test_gdrive_storage.py 2025-02-11 11:34:36 +00:00
erinhmclark 18666ff027 skip authenticated tests in test_gsheet_feeder.py 2025-02-11 11:28:24 +00:00
erinhmclark a69ac3e509 Fix file hash reference in S3 tests 2025-02-11 09:46:22 +00:00
erinhmclark c4bb667cec Merge branch 'load_modules' into add_module_tests 
# Conflicts:
#	src/auto_archiver/modules/s3_storage/s3_storage.py
#	src/auto_archiver/utils/gsheet.py
#	src/auto_archiver/utils/misc.py
 2025-02-10 16:17:08 +00:00
erinhmclark f311621e58 Small fixes. 
Add timestamp helper method.
 2025-02-10 15:57:42 +00:00
Patrick Robertson f3f6b92817 Implementation test cleanup 2025-02-10 12:43:21 +00:00
Patrick Robertson 74207d7821 Implementation tests for auto-archiver 2025-02-10 13:27:11 +01:00
erinhmclark e9ad1e1b85 Pass media to storage cdn_call 2025-02-06 22:01:55 +00:00
erinhmclark 266c7a14e6 Context related fixes, some more tests. 2025-02-06 16:53:00 +00:00
erinhmclark 67504a683e Merge branch 'load_modules' into add_module_tests 2025-02-06 10:13:37 +00:00
erinhmclark 5b0bad832f Updated test, test metadata 2025-02-06 10:11:56 +00:00
Patrick Robertson 6ab8fd2ee4 Tidy up setting modules as Orchestrator attributes on startup. 
Don't override the values in config['steps'] – the config should be left as is
 2025-02-06 10:20:05 +01:00
erinhmclark 52542812dc Merge tests from version with context. 2025-02-05 16:42:58 +00:00
Patrick Robertson 78e6418249 Unit tests for csv feeder + fix some bugs 2025-02-04 13:37:26 +01:00
Patrick Robertson c25d5cae84 Remove ArchivingContext completely 
Context for a specific url/item is now passed around via the metadata (metadata.set_context('key', 'val') and metadata.get_context('key', default='something')
The only other thing that was passed around in ArchivingContext was the storage info, which is already accessible now via self.config
 2025-01-30 17:50:54 +01:00
Patrick Robertson d76063c3f3 Fix unit tests 2025-01-30 16:46:53 +01:00