mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 198 Commity
13 Gałęzie
109 Tagi
19 MiB
Wykres commitów

154 Commity (17d2d14680d51edd5bc91a2012b4fb03e18cb4b9)

Autor SHA1 Wiadomość Data
Patrick Robertson 17d2d14680 Fix running 'cleanup' method on extractors that fail to start 2025-03-26 22:52:52 +04:00
Patrick Robertson d6be1ff84f Merge branch 'main' into timestamping_rewrite 2025-03-26 14:37:51 +04:00
Patrick Robertson a9fe959ea1 Fix unit tests for latest yt-dlp 
(Yt-dlp title is now truncated)
 2025-03-24 17:48:15 +04:00
Patrick Robertson 31fa7380f5 Fix up unit tests + issue when working with self-signed certs 2025-03-24 16:00:40 +04:00
Patrick Robertson 396ec03bae Tidy up unit tests further + make more non-download 2025-03-24 15:26:22 +04:00
Patrick Robertson e811196711 Ruff fixes 2025-03-24 15:10:46 +04:00
Patrick Robertson dfde6f1995 Merge main into timestamping_enricher 2025-03-24 15:09:29 +04:00
Patrick Robertson a066bf4ca9 Clean up comments 2025-03-21 14:47:50 +04:00
Patrick Robertson 14c56f4916 Provide better logs for screenshot enricher when auth is/isn't supported (cookies only) 2025-03-21 12:05:47 +04:00
Patrick Robertson 168dfb6254 Unit tests for url utils 2025-03-21 11:53:47 +04:00
Patrick Robertson e6c5705f70
Merge pull request #261 from bellingcat/wacz_separate_profile 
Wacz minor adjustments
 2025-03-20 15:51:56 +00:00
Erin Clark 613ba0c05d
Merge pull request #262 from bellingcat/generic_extractor_args 
Add flexible extractor_args to generic_extractor.py

This allows users to pass any of the options listed [here](https://github.com/yt-dlp/yt-dlp/blob/master/README.md#extractor-arguments) to yt-dlp extractor_args.

example usage:

```
generic_extractor:
  facebook_cookie:
  ...
  extractor_args:
    youtube:
      player_client: web,tv
    generic:
      is_live: true
```
 2025-03-20 15:38:20 +00:00
erinhmclark 54f53886ef Update tests for default config values 2025-03-20 14:57:26 +00:00
Patrick Robertson 0a5ba3385e Fix small bug in twitter dropin 
- previously the 'content' was being set to a json dump of the tweet, it should be set to full_text
 2025-03-20 18:55:22 +04:00
Patrick Robertson 6700250891 Add a test for checking module type on setup 2025-03-20 18:18:53 +04:00
Patrick Robertson 1e19ad77c6 Fix tests 2025-03-20 18:08:19 +04:00
Patrick Robertson f22af5e123 Tweak WACZ enricher docs + add comment on WACZ_ENABLE_DOCKER 2025-03-20 16:48:30 +04:00
erinhmclark 2921061fde Add flexible extractor_args to generic_extractor.py 2025-03-19 19:19:28 +00:00
Patrick Robertson 89e387030d Tests for suitable URLs for tikwm 2025-03-18 10:04:03 +00:00
Patrick Robertson 59b910ec30 Merge main 2025-03-17 10:05:11 +00:00
Patrick Robertson b21467c922 Fix ruff checks 2025-03-14 12:59:37 +00:00
Patrick Robertson 17ae75fb95 Ruff fixes 2025-03-14 12:38:12 +00:00
Patrick Robertson b8da7607e8 Merge branch 'main' into opentimestamps 2025-03-14 12:36:03 +00:00
erinhmclark 72f48f0147 Fix merge conflicts. 2025-03-14 12:11:24 +00:00
erinhmclark 846474a4e2 Merge branch 'main' into linting_etc 2025-03-14 10:50:13 +00:00
Patrick Robertson b908655cc8 Remove references to litecoin + several tidy-ups 2025-03-13 17:40:00 +00:00
Patrick Robertson 2e25e59fa6 Fix unit tests - make caplog checks more robust, having added a new logger/debug call 2025-03-13 16:07:49 +00:00
Patrick Robertson 10ceb7aa15 Move tikwm extractor into a droping for the generic extractor 2025-03-13 15:59:42 +00:00
erinhmclark 0bef78b0b4 Remove autouse property of mock_sleep. 2025-03-13 15:23:35 +00:00
Patrick Robertson 15222199d9 Add unit test for if one calendar fails 2025-03-13 14:45:38 +00:00
Patrick Robertson e7489ac4c4 Tidy up opentimestamps 
* Simplify
* Don't add fake (pending) attestations if the calendar urls all have issues
* Remove unnecessary configs
* Improve docs on upgrading + verifying
 2025-03-13 14:30:33 +00:00
erinhmclark 16012df30b Revert exception check in test. 2025-03-13 13:57:04 +00:00
erinhmclark 8673bc5979 Fix unused imports and include rule. 2025-03-13 13:55:31 +00:00
erinhmclark e76551ba22 Add documentation, pre-commit hook, more make commands and 2025-03-13 13:21:32 +00:00
erinhmclark 753c3c6214 Linting tests 2025-03-12 14:27:45 +00:00
Patrick Robertson 394b8b2dd1 Improvements to opentimestamps enricher - make OTS file a sub-file of original media 2025-03-12 11:45:13 +00:00
erinhmclark 79f576be1d Run fix on tests. 2025-03-12 10:38:16 +00:00
Patrick Robertson 1423c10363 Finish off timestamping module 2025-03-12 10:24:57 +00:00
erinhmclark 8ca7698fa0 Move Makefile and fix import error with unused import. 2025-03-11 19:58:02 +00:00
Patrick Robertson 28041d94d9 Add unit tests for opentimestamps enricher 2025-03-11 17:33:54 +00:00
Patrick Robertson 89ee6f19b6 List out all valid TSAs + option for users to allow self signed if they want 2025-03-11 16:12:13 +00:00
Patrick Robertson 294033f156 Fix bug ordering tsr that only have one cert + more unit tests 2025-03-11 15:44:04 +00:00
Patrick Robertson 2ffe124d95 Add unit test for invalid digicert tsrs 2025-03-11 11:13:36 +00:00
Patrick Robertson 1db8be91db Improved unit tests for timestamping 2025-03-11 11:08:52 +00:00
erinhmclark 441f341139 Merge branch 'main' into linting_etc 
# Conflicts:
#	src/auto_archiver/core/consts.py
#	src/auto_archiver/core/orchestrator.py
#	src/auto_archiver/core/storage.py
#	src/auto_archiver/modules/local_storage/local_storage.py
#	src/auto_archiver/modules/s3_storage/s3_storage.py
#	tests/storages/test_S3_storage.py
#	tests/storages/test_local_storage.py
#	tests/storages/test_storage_base.py
 2025-03-11 10:39:47 +00:00
Patrick Robertson e2442b2f6b
Merge pull request #243 from bellingcat/fix-long-path-names 
Unit tests for storage types + fix storage too long issues for local storage
 2025-03-11 10:05:09 +00:00
Patrick Robertson 3f6acc0917 fully working timestamping enricher 2025-03-11 10:04:46 +00:00
erinhmclark 85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
Patrick Robertson 2b91dc9514 Fix up unit tests 2025-03-10 16:51:16 +00:00
Patrick Robertson 770f4c8a3d Refactoring of storage code: 
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
 2025-03-10 16:39:48 +00:00