mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
1 461 Commity
11 Gałęzie
113 Tagi
22 MiB
Wykres commitów

705 Commity (dev)

Autor SHA1 Wiadomość Data
msramalho fdbe96f2e4
vk and reddit should work without credentials but log the error 2025-06-10 16:44:14 +01:00
msramalho 773fa82f06
introduces reddit dropin 2025-06-10 16:31:19 +01:00
msramalho ef0e909a72
extractor to auto detect best quality 2025-06-10 16:29:35 +01:00
msramalho 6bbc7fb47a
improves antibot flow and makes auth_wall detection optional 2025-06-10 16:29:07 +01:00
msramalho 809b8c7749
default dropin introduced 2025-06-10 16:14:42 +01:00
msramalho 6d82655cc4
manifest improvement for antibot 2025-06-10 16:14:34 +01:00
msramalho 6bd493a791
dropin with new ytdlp feature and helper method 2025-06-10 16:11:55 +01:00
msramalho 287e823f43
improves twitter URL cleaning and introduces another bestquality check 2025-06-10 16:09:38 +01:00
msramalho c815488daa
adds new URLs to ignore 2025-06-10 15:44:52 +01:00
msramalho 6f02493ff1
adds clips extraction to VK, though generic_extractor should still be run for those 2025-06-08 14:36:55 +01:00
msramalho 1f2d637928
minor improvements 2025-06-08 14:16:21 +01:00
msramalho 18cc05a2fe
allows auth_for_site to receive do.main directly 2025-06-08 14:16:12 +01:00
msramalho c96fd71f35
minor cleanup 2025-06-07 20:06:53 +01:00
msramalho d13a5ef003
adds tests in minor improvements 2025-06-07 19:58:18 +01:00
msramalho 48c1ab3c1f
doc improvements 2025-06-07 19:14:16 +01:00
msramalho b2ee42ee95
adds the first antibot dropin: VKontakte 2025-06-07 19:10:01 +01:00
msramalho 07ff5baf07
adds Dropin flexible integration for antibot 2025-06-07 19:09:37 +01:00
msramalho d202d79e0f
lint 2025-06-07 19:06:14 +01:00
msramalho e2e6490b49
minimal changes 2025-06-07 18:15:21 +01:00
msramalho 952487da30
adds missing bin dependency 2025-06-07 18:14:42 +01:00
msramalho c7a84bc97a
generalizes ydl info to filename method for reusing 2025-06-07 18:14:08 +01:00
msramalho 8a897cf601
minimal changes: standard naming 2025-06-04 15:06:08 +01:00
Miguel Sozinho Ramalho 14c8af5cc8
Merge pull request #310 from djhmateer/waczscreenshot bug fix 
counter_screenshots to counter_warc_files in wacz_extractor so don't …
 2025-06-04 15:01:12 +01:00
msramalho 264ba82ea0
finish removing screenshot_enricher references 2025-06-04 14:31:07 +01:00
msramalho 5f68c151a0
removes webdriver utils used by screenshot enricher 2025-06-04 14:17:19 +01:00
msramalho bc8cf2fb29
minor TODO 2025-06-04 14:10:19 +01:00
msramalho e6f3826a3a
dropping screenshot enricher 2025-06-04 12:08:59 +01:00
msramalho e5a78a5d06
antibot can be used out of the box 2025-06-04 12:01:42 +01:00
msramalho 258fb4faaf
visual HTML preview improvements 2025-06-04 12:00:40 +01:00
msramalho 22408e2a98
adds test for antibot 2025-06-04 11:59:59 +01:00
msramalho 378b1a6d22
expand S3 objects content type for better preview results in non-latin languages 2025-06-04 11:53:41 +01:00
msramalho d130c1b3fa
WIP attempt at ytdlp impersonation 2025-06-04 11:53:18 +01:00
msramalho cbd189c97d
general cleanup 2025-06-04 11:53:01 +01:00
msramalho d2e8f1a512
introduces antibot step with seleniumbase 2025-06-04 11:20:46 +01:00
Dave Mateer c772082f0e counter_screenshots to counter_warc_files in wacz_extractor so don't get error about add mulitple items with same id. 2025-06-03 12:34:41 +01:00
msramalho ee68f3efee
Merge remote-tracking branch 'origin/main' into feat/seleniumbase 2025-06-03 11:05:16 +01:00
Miguel Sozinho Ramalho 6735fa890b
v1.0.1 dependency updates, generic extractor improvements (#307) 
* wacz: allow exceptional cases where more than one resource image is available

* improves generic extractor edge-cases and yt-dlp updates

* REMOVES vk_extractor until further notice

* bumps browsertrix in docker image

* npm version bump on scripts/settings

* poetry updates

* Changed log level on gsheet_feeder_db started from warning to info (#301)

* closes 305 and further fixes finding local downloads from uncommon ytdlp extractors

* use ffmpeg -bitexact to reduce duplicate content storing

* formatting

* adds yt-dlp curl-cffi

* version bump

* linting

---------

Co-authored-by: Dave Mateer <davemateer@gmail.com>
 2025-06-02 20:57:12 +01:00
Dave Mateer 48be13fb2a
catch for if self.comments are true but no actual comments in video (#303) 
* catch for if self.comments are true but no actual comments in video

* simplifies check code

---------

Co-authored-by: Miguel Sozinho Ramalho <19508417+msramalho@users.noreply.github.com>
 2025-06-02 13:02:19 +01:00
msramalho 5cf640af8a
experiments with seleniumbase 2025-04-28 11:08:00 +01:00
erinhmclark 144adaad5b Only return success for instagram_tbot_extractor.py with content. 2025-03-31 14:14:36 +01:00
Erin Clark c7c7eb00a1
Merge pull request #286 from bellingcat/version_comparison 
Small code fixes and GH Actions cache
 2025-03-31 12:40:42 +01:00
erinhmclark 7e4ba62918 Small code change 2025-03-31 12:05:39 +01:00
erinhmclark 8940580638 Add poetry cache clear, and small code change 2025-03-31 11:41:26 +01:00
erinhmclark a590647279 Small code tidy to trigger tests. 2025-03-31 11:23:49 +01:00
Erin Clark 8685b6bf13
Merge pull request #285 from bellingcat/fix-ubuntu-22 
[WIP] Change order of poetry install - in case this fixes core tests
 2025-03-28 15:38:03 +00:00
Patrick Robertson fd540bd03a Code change to trigger tests 2025-03-28 18:29:59 +04:00
erinhmclark 68992025b0 Update version comparison. 2025-03-28 14:29:44 +00:00
Erin Clark 96efdcbba1
Merge pull request #281 from bellingcat/add_inst_api_script 
Add InstagrAPI server script to be used with the Instagram API Extractor.
 2025-03-28 13:58:37 +00:00
erinhmclark 5e7c57650b Update "default" to "auto" for clarity, update docs 2025-03-28 13:16:16 +00:00
erinhmclark b5dc1854a2 Merge branch 'main' into feat/yt-dlp-pots 2025-03-28 10:42:24 +00:00
Patrick Robertson 25f1f5dc93
Merge pull request #279 from bellingcat/telethon_tweaks 
Fix calling extractor.cleanup (fixes telethon issue) + tidy up telethon extractor session file naming
 2025-03-28 14:13:26 +04:00
erinhmclark 00b29db390 Update documentaion for instagrapi api 2025-03-28 00:41:35 +00:00
Patrick Robertson a448e2532c Code tweak for clarity 2025-03-27 15:20:52 +04:00
Patrick Robertson b7949a489f Simplify telethon unit tests for CI (don't use TestExtractorBase - it causes loading issues) 2025-03-26 23:51:21 +04:00
Patrick Robertson 95ea9fb231 Telethon unit tests + tidyup 2025-03-26 22:53:27 +04:00
Patrick Robertson 17d2d14680 Fix running 'cleanup' method on extractors that fail to start 2025-03-26 22:52:52 +04:00
erinhmclark 2cad5edea8 Fix default config 2025-03-26 17:33:00 +00:00
Patrick Robertson 580de88366 Set the new session filename *before* copying 
Fixes a potential bug whereby if the copy fails for some reason, the 'cleanup' command would remove the original session file
 2025-03-26 21:32:23 +04:00
erinhmclark 093ce34a6a Ruff format. 2025-03-26 17:02:20 +00:00
erinhmclark 7872d9356c Merge branch 'main' into feat/yt-dlp-pots 2025-03-26 17:00:38 +00:00
erinhmclark ae523eb06f Udpate PO token generation script method 2025-03-26 16:45:29 +00:00
erinhmclark d87c0dc3a9 Implement update for pot plugin. 2025-03-26 16:02:29 +00:00
Patrick Robertson 20bc80b9ef Slightly more consistent/tidier naming for the session files 
Don't add/remove .session from name, keep the file name without .session at all times
 2025-03-26 15:57:11 +04:00
Patrick Robertson 76e90dd23a Small code tidy ups 2025-03-26 15:34:33 +04:00
Patrick Robertson 0073a08525 Update manifest dependencies to remove tsp_client et al. 2025-03-26 14:57:55 +04:00
Patrick Robertson d6be1ff84f Merge branch 'main' into timestamping_rewrite 2025-03-26 14:37:51 +04:00
erinhmclark 633290a9cc Update for pot providers list 2025-03-25 18:27:06 +00:00
erinhmclark 040a864d5c Merge branch 'refs/heads/main' into feat/yt-dlp-pots 
# Conflicts:
#	poetry.lock
 2025-03-25 18:26:43 +00:00
erinhmclark b4c33318c4 Merge branch 'main' into feat/yt-dlp-pots 
# Conflicts:
#	src/auto_archiver/modules/generic_extractor/__manifest__.py
#	tests/test_modules.py
 2025-03-25 15:16:31 +00:00
Patrick Robertson beb7f3893d Add comments/notes to WACZ enricher about browser profiles 2025-03-24 17:39:47 +04:00
Patrick Robertson 3c4625d708 Further ruff tweaks 2025-03-24 16:39:59 +04:00
Patrick Robertson 31fa7380f5 Fix up unit tests + issue when working with self-signed certs 2025-03-24 16:00:40 +04:00
Patrick Robertson 396ec03bae Tidy up unit tests further + make more non-download 2025-03-24 15:26:22 +04:00
Patrick Robertson dfde6f1995 Merge main into timestamping_enricher 2025-03-24 15:09:29 +04:00
Patrick Robertson c980500978 Actually restart AA after updating yt-dlp. 
A simple 'importlib.reload()' doesn't take into account all imports
 2025-03-24 14:33:59 +04:00
Patrick Robertson aacb874b56 removeprefix for www. is required here 2025-03-21 12:23:45 +04:00
Patrick Robertson 4b5a8c0199 Add warning *inside* instagram_extractor that it's not actively maintained 2025-03-21 12:09:58 +04:00
Patrick Robertson 14c56f4916 Provide better logs for screenshot enricher when auth is/isn't supported (cookies only) 2025-03-21 12:05:47 +04:00
Patrick Robertson 5b131996c6 Add return type for auth_for_site 2025-03-21 11:55:12 +04:00
Patrick Robertson 168dfb6254 Unit tests for url utils 2025-03-21 11:53:47 +04:00
Patrick Robertson 42e16aebd6
Merge pull request #255 from bellingcat/autogenerate_services_account 
Script to auto-generate a service account
 2025-03-20 18:00:45 +00:00
Patrick Robertson e6c5705f70
Merge pull request #261 from bellingcat/wacz_separate_profile 
Wacz minor adjustments
 2025-03-20 15:51:56 +00:00
Erin Clark 613ba0c05d
Merge pull request #262 from bellingcat/generic_extractor_args 
Add flexible extractor_args to generic_extractor.py

This allows users to pass any of the options listed [here](https://github.com/yt-dlp/yt-dlp/blob/master/README.md#extractor-arguments) to yt-dlp extractor_args.

example usage:

```
generic_extractor:
  facebook_cookie:
  ...
  extractor_args:
    youtube:
      player_client: web,tv
    generic:
      is_live: true
```
 2025-03-20 15:38:20 +00:00
Patrick Robertson 0a5ba3385e Fix small bug in twitter dropin 
- previously the 'content' was being set to a json dump of the tweet, it should be set to full_text
 2025-03-20 18:55:22 +04:00
Patrick Robertson 034857075d Merge branch 'main' into wrong_steps 2025-03-20 18:44:19 +04:00
Patrick Robertson 5e5e1c43a1 When loading modules, check they have been added to the right 'step' in the config 
Fixes an issue seen on discord where a user accidentally set up metadata_enricher under 'extractors'
 2025-03-20 18:09:26 +04:00
Patrick Robertson f22af5e123 Tweak WACZ enricher docs + add comment on WACZ_ENABLE_DOCKER 2025-03-20 16:48:30 +04:00
erinhmclark 2921061fde Add flexible extractor_args to generic_extractor.py 2025-03-19 19:19:28 +00:00
Patrick Robertson e531906d73 Create an independent profile file for each wacz_extractor_enricher instance 2025-03-19 18:08:24 +04:00
Patrick Robertson 244341d22c Skip check for 'docker' bin dependency if already running in docker 2025-03-19 18:08:04 +04:00
Patrick Robertson 488675056b Download generate_google_services.sh script from GH - it's not packaged with the app 2025-03-19 15:52:39 +04:00
erinhmclark fc6946f78a Run format. 2025-03-18 21:43:18 +00:00
erinhmclark 2fdf6b7564 Update generic_extractor.py for general/ youtube extraction. 2025-03-18 21:33:21 +00:00
erinhmclark a577228465 Update generic_extractor.py for general/ youtube extraction. 2025-03-18 21:10:06 +00:00
erinhmclark ba9d67e4bb Merge branch 'main' into feat/yt-dlp-pots 2025-03-18 20:10:38 +00:00
erinhmclark c4e63ebd8c Add conditional check to setup bgutils token generation script. 
TODO: Update tests
 2025-03-18 14:54:57 +00:00
Miguel Sozinho Ramalho f6863b8eb2
Update src/auto_archiver/modules/gsheet_feeder_db/__manifest__.py 2025-03-18 14:10:47 +00:00
erinhmclark cb632723bd Add scripts to pull only /server/ section of pots generator, adn only install at runtime. 2025-03-18 13:47:01 +00:00
erinhmclark 0c892f3cf1 Temp fix for tests by setting path in manifest. 2025-03-18 11:44:08 +00:00
Patrick Robertson d03ecdb037 Standardise parse dates to get_datetime_from_str 2025-03-18 10:22:58 +00:00