mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
719 Commity
15 Gałęzie
109 Tagi
19 MiB
Wykres commitów

22 Commity (fd2e7f973b75d7bc5fe8cf8956464da283a34ff3)

Autor SHA1 Wiadomość Data
Patrick Robertson fd2e7f973b Further tidy-ups, also adds some ytdlp utils to 'utils' 2025-01-20 16:31:28 +01:00
Patrick Robertson befc92deb4 Further unit test tidy ups 2025-01-17 17:29:13 +01:00
Patrick Robertson d4893ee05e Fix unit tests for base_archiver->generic_archiver rename 2025-01-17 17:08:00 +01:00
Patrick Robertson 17c1c9c360 Fix up core unit tests when a twitter api key isn't provided 2025-01-17 12:02:38 +01:00
Patrick Robertson 394bcd8d47 Further refactoring of youtubedl_archiver->base_archiver 
* Keep twitter_api_archiver
* Remove unit tests for obsolete archivers
* Guess filename of media using the 'Content-Type' header
* Add mechanism to run 'expensive' tests last (see conftest.py) and also flag expensive tests to fail straight off (pytest.mark.incremental)
 2025-01-17 11:56:08 +01:00
Patrick Robertson 3168bed0d9 Add (skipped) test for twitter extraction with youtubedlp 2025-01-15 19:00:57 +01:00
Patrick Robertson 5626bba815 Add test on bluesky and note on why it doesn't work 2025-01-15 18:31:20 +01:00
Patrick Robertson 74cf1f5f23 Merge branch 'main' into youtubedlp-rewrite 2025-01-15 17:47:23 +01:00
Patrick Robertson 4f2b9baa73 refactor youtubedlp archiver to work for all valid websites 
1. Extract more metadata
2. Better extract thumbnail
3. Setup framework for specific sites to provide more granular metadata processing
 2025-01-15 17:46:47 +01:00
Patrick Robertson 20726c1116 Remove tiktok-downloader - getting info is broken 
TODO: switch to using youtube-dlp
 2025-01-14 17:40:45 +01:00
Patrick Robertson 6f10270baf Remove unittest and switch to pytest fully 2025-01-14 16:28:39 +01:00
Patrick Robertson 528b78db85 Flag tombstone tweets for twitter_syndication method 2025-01-13 18:17:24 +01:00
Patrick Robertson 57eacdc24a Merge branch 'main' into feat/unittest 2025-01-13 18:06:55 +01:00
Patrick Robertson bbef80de4c Add unit tests for html_formatter, csv_db 2025-01-13 17:58:10 +01:00
Patrick Robertson 63973e2ce7 switch to pytest and pytest-recording 2025-01-13 16:23:20 +01:00
Patrick Robertson e2bc84ccb9 Merge branch 'main' into feat/unittest 2025-01-13 13:15:13 +01:00
Patrick Robertson 3546d4ad79 Fix 'download_syndication' method for tweet archiving (now requires a token) 
Plus add in unit tests for token generation + download syndication
 2025-01-12 12:55:00 +01:00
Patrick Robertson c932fb7416 Improved logging when an invalid/deleted tweet is attempted to be downloaded 
Plus: unit tests for non-existent tweet + invalid tweet ID
 2025-01-12 12:00:45 +01:00
Patrick Robertson 9dc4eb35de Switch to pytest and use vcr for request storing 2025-01-08 11:25:13 +01:00
Patrick Robertson 8c044c15f0 Add base test class for archivers with boilerplate code 
Plus: create test class for twitter archiver. Currently WIP
 2025-01-08 10:38:56 +01:00
Miguel Sozinho Ramalho a697f0a212
adds an unauthenticated Bluesky archiver (#160) 
* adds a TODO for next code iterations

* implements bsky archiver

* adds new archiver to example orchestration file

* Fix downloading media for posts with multiple images

(Images are stored in media/images)

* Setup a basic framework for unit tests

Use 'python -m unittest' from the project root to run

---------

Co-authored-by: Patrick Robertson <robertson.patrick@gmail.com>
 2025-01-07 10:28:07 +00:00
Patrick Robertson 30d423c8e6 Setup a basic framework for unit tests 
Use 'python -m unittest' from the project root to run
 2024-12-31 14:29:52 +01:00