mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 157 Commity
12 Gałęzie
113 Tagi
22 MiB
Wykres commitów

134 Commity (scoop)

Autor SHA1 Wiadomość Data
Patrick Robertson a066bf4ca9 Clean up comments 2025-03-21 14:47:50 +04:00
Patrick Robertson 14c56f4916 Provide better logs for screenshot enricher when auth is/isn't supported (cookies only) 2025-03-21 12:05:47 +04:00
Patrick Robertson 168dfb6254 Unit tests for url utils 2025-03-21 11:53:47 +04:00
Patrick Robertson e6c5705f70
Merge pull request #261 from bellingcat/wacz_separate_profile 
Wacz minor adjustments
 2025-03-20 15:51:56 +00:00
Erin Clark 613ba0c05d
Merge pull request #262 from bellingcat/generic_extractor_args 
Add flexible extractor_args to generic_extractor.py

This allows users to pass any of the options listed [here](https://github.com/yt-dlp/yt-dlp/blob/master/README.md#extractor-arguments) to yt-dlp extractor_args.

example usage:

```
generic_extractor:
  facebook_cookie:
  ...
  extractor_args:
    youtube:
      player_client: web,tv
    generic:
      is_live: true
```
 2025-03-20 15:38:20 +00:00
erinhmclark 54f53886ef Update tests for default config values 2025-03-20 14:57:26 +00:00
Patrick Robertson 0a5ba3385e Fix small bug in twitter dropin 
- previously the 'content' was being set to a json dump of the tweet, it should be set to full_text
 2025-03-20 18:55:22 +04:00
Patrick Robertson 6700250891 Add a test for checking module type on setup 2025-03-20 18:18:53 +04:00
Patrick Robertson 1e19ad77c6 Fix tests 2025-03-20 18:08:19 +04:00
Patrick Robertson f22af5e123 Tweak WACZ enricher docs + add comment on WACZ_ENABLE_DOCKER 2025-03-20 16:48:30 +04:00
erinhmclark 2921061fde Add flexible extractor_args to generic_extractor.py 2025-03-19 19:19:28 +00:00
Patrick Robertson 89e387030d Tests for suitable URLs for tikwm 2025-03-18 10:04:03 +00:00
Patrick Robertson 59b910ec30 Merge main 2025-03-17 10:05:11 +00:00
Patrick Robertson b21467c922 Fix ruff checks 2025-03-14 12:59:37 +00:00
Patrick Robertson 17ae75fb95 Ruff fixes 2025-03-14 12:38:12 +00:00
Patrick Robertson b8da7607e8 Merge branch 'main' into opentimestamps 2025-03-14 12:36:03 +00:00
erinhmclark 72f48f0147 Fix merge conflicts. 2025-03-14 12:11:24 +00:00
erinhmclark 846474a4e2 Merge branch 'main' into linting_etc 2025-03-14 10:50:13 +00:00
Patrick Robertson b908655cc8 Remove references to litecoin + several tidy-ups 2025-03-13 17:40:00 +00:00
Patrick Robertson 2e25e59fa6 Fix unit tests - make caplog checks more robust, having added a new logger/debug call 2025-03-13 16:07:49 +00:00
Patrick Robertson 10ceb7aa15 Move tikwm extractor into a droping for the generic extractor 2025-03-13 15:59:42 +00:00
erinhmclark 0bef78b0b4 Remove autouse property of mock_sleep. 2025-03-13 15:23:35 +00:00
Patrick Robertson 15222199d9 Add unit test for if one calendar fails 2025-03-13 14:45:38 +00:00
Patrick Robertson e7489ac4c4 Tidy up opentimestamps 
* Simplify
* Don't add fake (pending) attestations if the calendar urls all have issues
* Remove unnecessary configs
* Improve docs on upgrading + verifying
 2025-03-13 14:30:33 +00:00
erinhmclark 16012df30b Revert exception check in test. 2025-03-13 13:57:04 +00:00
erinhmclark 8673bc5979 Fix unused imports and include rule. 2025-03-13 13:55:31 +00:00
erinhmclark e76551ba22 Add documentation, pre-commit hook, more make commands and 2025-03-13 13:21:32 +00:00
erinhmclark 753c3c6214 Linting tests 2025-03-12 14:27:45 +00:00
Patrick Robertson 394b8b2dd1 Improvements to opentimestamps enricher - make OTS file a sub-file of original media 2025-03-12 11:45:13 +00:00
erinhmclark 79f576be1d Run fix on tests. 2025-03-12 10:38:16 +00:00
Patrick Robertson 1423c10363 Finish off timestamping module 2025-03-12 10:24:57 +00:00
erinhmclark 8ca7698fa0 Move Makefile and fix import error with unused import. 2025-03-11 19:58:02 +00:00
Patrick Robertson 28041d94d9 Add unit tests for opentimestamps enricher 2025-03-11 17:33:54 +00:00
erinhmclark 441f341139 Merge branch 'main' into linting_etc 
# Conflicts:
#	src/auto_archiver/core/consts.py
#	src/auto_archiver/core/orchestrator.py
#	src/auto_archiver/core/storage.py
#	src/auto_archiver/modules/local_storage/local_storage.py
#	src/auto_archiver/modules/s3_storage/s3_storage.py
#	tests/storages/test_S3_storage.py
#	tests/storages/test_local_storage.py
#	tests/storages/test_storage_base.py
 2025-03-11 10:39:47 +00:00
Patrick Robertson e2442b2f6b
Merge pull request #243 from bellingcat/fix-long-path-names 
Unit tests for storage types + fix storage too long issues for local storage
 2025-03-11 10:05:09 +00:00
erinhmclark 85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
Patrick Robertson 2b91dc9514 Fix up unit tests 2025-03-10 16:51:16 +00:00
Patrick Robertson 770f4c8a3d Refactoring of storage code: 
1. Fix some bugs in local_storage
2. Refactor unit tests to not set Media.key explicitly (unless it's well-known beforehand, which it isn't)
3. Limit length of URL for 'url' type path_generator
4. Throw an error if 'save_to' of local storage is too long
5. A few other tidyups
 2025-03-10 16:39:48 +00:00
Miguel Sozinho Ramalho 58bd38e292
Adds new extractor for tiktok via unofficial API (#237) 
* minor update to defaults in api_db

* readme typo

* adds and tests new tikwm tiktok downloader

* addresses PR comments
 2025-03-10 11:56:45 +00:00
Patrick Robertson e89a8da3b4 Unit tests for storage types + fix storage too long issues for local storage 2025-03-10 11:30:15 +00:00
Patrick Robertson be513e95aa Merge branch 'main' into merge_modules 2025-03-07 16:19:51 +00:00
erinhmclark fa1e65f54c Fix instagram_extractor.py typo, add warning to docs, and add basic regex test. 2025-03-06 16:25:38 +00:00
erinhmclark b9c2f98f46 Update Atlos tests 2025-03-05 21:24:38 +00:00
erinhmclark d775e4612e Update gsheet_feeder references in tests. 2025-03-04 14:06:46 +00:00
erinhmclark 077b56c150 Merge GSheet Feeder and Database. 2025-03-04 14:05:19 +00:00
erinhmclark a705a78632 Fix instagram_extractor.py typo in config value. 2025-03-03 21:06:09 +00:00
erinhmclark 4280791f07 Fix mocking in test_wayback_enricher.py. 2025-02-27 11:25:58 +00:00
erinhmclark 8124bb831d Merge branch 'main' into small_issues 
# Conflicts:
#	src/auto_archiver/core/base_module.py
#	src/auto_archiver/utils/misc.py
 2025-02-26 13:19:49 +00:00
erinhmclark b2e654aef9 Remove context manager from test_pdq_hash_enricher.py 2025-02-26 12:57:33 +00:00
erinhmclark cf1219f798 Add text content into gsheet. 2025-02-25 20:06:44 +00:00