mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 386 Commity
10 Gałęzie
110 Tagi
21 MiB
Wykres commitów

57 Commity (dev)

Autor SHA1 Wiadomość Data
msramalho 264ba82ea0
finish removing screenshot_enricher references 2025-06-04 14:31:07 +01:00
Miguel Sozinho Ramalho 6735fa890b
v1.0.1 dependency updates, generic extractor improvements (#307) 
* wacz: allow exceptional cases where more than one resource image is available

* improves generic extractor edge-cases and yt-dlp updates

* REMOVES vk_extractor until further notice

* bumps browsertrix in docker image

* npm version bump on scripts/settings

* poetry updates

* Changed log level on gsheet_feeder_db started from warning to info (#301)

* closes 305 and further fixes finding local downloads from uncommon ytdlp extractors

* use ffmpeg -bitexact to reduce duplicate content storing

* formatting

* adds yt-dlp curl-cffi

* version bump

* linting

---------

Co-authored-by: Dave Mateer <davemateer@gmail.com>
 2025-06-02 20:57:12 +01:00
Patrick Robertson 31fa7380f5 Fix up unit tests + issue when working with self-signed certs 2025-03-24 16:00:40 +04:00
Patrick Robertson 396ec03bae Tidy up unit tests further + make more non-download 2025-03-24 15:26:22 +04:00
Patrick Robertson e811196711 Ruff fixes 2025-03-24 15:10:46 +04:00
Patrick Robertson dfde6f1995 Merge main into timestamping_enricher 2025-03-24 15:09:29 +04:00
Patrick Robertson 14c56f4916 Provide better logs for screenshot enricher when auth is/isn't supported (cookies only) 2025-03-21 12:05:47 +04:00
Patrick Robertson f22af5e123 Tweak WACZ enricher docs + add comment on WACZ_ENABLE_DOCKER 2025-03-20 16:48:30 +04:00
Patrick Robertson b21467c922 Fix ruff checks 2025-03-14 12:59:37 +00:00
Patrick Robertson 17ae75fb95 Ruff fixes 2025-03-14 12:38:12 +00:00
Patrick Robertson b8da7607e8 Merge branch 'main' into opentimestamps 2025-03-14 12:36:03 +00:00
Patrick Robertson b908655cc8 Remove references to litecoin + several tidy-ups 2025-03-13 17:40:00 +00:00
erinhmclark 0bef78b0b4 Remove autouse property of mock_sleep. 2025-03-13 15:23:35 +00:00
Patrick Robertson 15222199d9 Add unit test for if one calendar fails 2025-03-13 14:45:38 +00:00
Patrick Robertson e7489ac4c4 Tidy up opentimestamps 
* Simplify
* Don't add fake (pending) attestations if the calendar urls all have issues
* Remove unnecessary configs
* Improve docs on upgrading + verifying
 2025-03-13 14:30:33 +00:00
erinhmclark 753c3c6214 Linting tests 2025-03-12 14:27:45 +00:00
Patrick Robertson 394b8b2dd1 Improvements to opentimestamps enricher - make OTS file a sub-file of original media 2025-03-12 11:45:13 +00:00
erinhmclark 79f576be1d Run fix on tests. 2025-03-12 10:38:16 +00:00
Patrick Robertson 1423c10363 Finish off timestamping module 2025-03-12 10:24:57 +00:00
Patrick Robertson 28041d94d9 Add unit tests for opentimestamps enricher 2025-03-11 17:33:54 +00:00
Patrick Robertson 89ee6f19b6 List out all valid TSAs + option for users to allow self signed if they want 2025-03-11 16:12:13 +00:00
Patrick Robertson 294033f156 Fix bug ordering tsr that only have one cert + more unit tests 2025-03-11 15:44:04 +00:00
Patrick Robertson 2ffe124d95 Add unit test for invalid digicert tsrs 2025-03-11 11:13:36 +00:00
Patrick Robertson 1db8be91db Improved unit tests for timestamping 2025-03-11 11:08:52 +00:00
erinhmclark 441f341139 Merge branch 'main' into linting_etc 
# Conflicts:
#	src/auto_archiver/core/consts.py
#	src/auto_archiver/core/orchestrator.py
#	src/auto_archiver/core/storage.py
#	src/auto_archiver/modules/local_storage/local_storage.py
#	src/auto_archiver/modules/s3_storage/s3_storage.py
#	tests/storages/test_S3_storage.py
#	tests/storages/test_local_storage.py
#	tests/storages/test_storage_base.py
 2025-03-11 10:39:47 +00:00
Patrick Robertson 3f6acc0917 fully working timestamping enricher 2025-03-11 10:04:46 +00:00
erinhmclark 85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
Patrick Robertson 770f4c8a3d Refactoring of storage code: 
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
 2025-03-10 16:39:48 +00:00
erinhmclark 4280791f07 Fix mocking in test_wayback_enricher.py. 2025-02-27 11:25:58 +00:00
erinhmclark 8124bb831d Merge branch 'main' into small_issues 
# Conflicts:
#	src/auto_archiver/core/base_module.py
#	src/auto_archiver/utils/misc.py
 2025-02-26 13:19:49 +00:00
erinhmclark b2e654aef9 Remove context manager from test_pdq_hash_enricher.py 2025-02-26 12:57:33 +00:00
Patrick Robertson afc117a229 Get downloading certs working 2025-02-26 09:33:56 +00:00
Patrick Robertson 4dcb77c29f Merge branch 'main' into timestamping_rewrite 2025-02-25 17:10:55 +00:00
Patrick Robertson 898faf6fe4 Further WIP - currently working on verify_signed 2025-02-25 12:08:08 +00:00
Patrick Robertson 01bf88a695 Merge branch 'main' into timestamping_rewrite 2025-02-24 12:03:14 +00:00
Patrick Robertson 4174285898 Fix unit tests 2025-02-20 13:18:06 +00:00
Patrick Robertson 5211c5de18
Merge pull request #210 from bellingcat/logger_fix 
Fix issue #200 + Refactor _LAZY_LOADED_MODULES
 2025-02-19 15:11:42 +00:00
erinhmclark 47a634fc63 Add WACZ, Wayback and local storage tests. 2025-02-19 13:14:08 +00:00
Patrick Robertson a9802dd004 Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules 2025-02-19 12:25:35 +00:00
erinhmclark 10a5ad62b8 Include Atlos tests, metadata fixture. 2025-02-19 09:18:41 +00:00
erinhmclark f0fd9bf445 Updates tests to use pytest-mock. 2025-02-18 23:32:03 +00:00
erinhmclark 7b88df72cb Update test_metadata_enricher.py 2025-02-18 19:46:57 +00:00
erinhmclark 71b41dd901 Remove accidental path, yet again. 2025-02-14 10:05:32 +00:00
erinhmclark 319c1e8f92 Add more tests. 2025-02-14 09:48:37 +00:00
erinhmclark cbe98c729d Enricher tests 2025-02-12 19:32:40 +00:00
erinhmclark d9d936c2ca Thumbnail enricher fix seconds to minutes. 2025-02-12 12:22:27 +00:00
Patrick Robertson d0c379a3ba WIP - timestamping enricher 2025-02-11 18:18:19 +00:00
Patrick Robertson 3163cb793a Fix timestamping enricher for new module structure (temp paths) 2025-02-11 15:26:40 +00:00
Patrick Robertson 7bb4d68a22 Merge branch 'load_modules' into timestamping_rewrite 2025-02-11 15:21:31 +00:00
erinhmclark 266c7a14e6 Context related fixes, some more tests. 2025-02-06 16:53:00 +00:00