mirror
/
auto-archiver
kopia lustrzana https://github.com/bellingcat/auto-archiver
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność
898 Commity
7 Gałęzie
113 Tagi
22 MiB
Wykres commitów

898 Commity (222a94563f1f9ea5754a521aa391d033d260f47d)

Autor SHA1 Wiadomość Data
Patrick Robertson 222a94563f WIP: Docs tidyups+add howto on logging and authentication 
(Authentication is WIP)
 2025-02-19 10:37:04 +00:00
Patrick Robertson 3c543a3a6a
Various fixes for issues with new architecture (#208) 
* Add formatters to the TOC - fixes #204

* Add 'steps' settings to the example YAML in the docs. Fixes #206

* Improved docs on authentication architecture

* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending

* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)

* Rename the command line entrypoint to _command_line_run

Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)

* Fix unit tests to use now code-entry points

* Version bump

* Move iterating of generator up to __main__

* Breakpoint

* two minor fixes

* Fix unit tests + add new '__main__' entry point implementation test

* Skip youtube tests if running on CI. Should still run them locally

* Fix full implementation run on GH actions

* Fix skipif test for GH Actions CI

* Add skipifs for truth - it blocks GH:

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
 2025-02-18 19:10:09 +00:00
Patrick Robertson 6d43bc7d4d
Fix generator programmatic setup (#197) 
* Fix returning a generator of a generator

* Move download test test to pytest.mark.download
 2025-02-15 17:36:44 +00:00
Miguel Sozinho Ramalho 9297697ef5
makes orchestrator.run return the results to allow for code integration (#196) 2025-02-15 12:41:26 +00:00
Miguel Sozinho Ramalho 5614af3f63
removes fixed oscrypto dependency, it blocked pypi publishing (#195) 
* disables tests on ubuntu-latest

* drops fixed oscrypto version for git commit

* version bump
 2025-02-14 10:51:56 +00:00
Miguel Sozinho Ramalho 27f9287b65
markdown fixes 2025-02-12 17:37:36 +00:00
Patrick Robertson d849678137
Merge pull request #193 from bellingcat/links 
Fix links to docs
 2025-02-12 13:11:59 +01:00
erinhmclark da267f20d7 Update screenshot refs 2025-02-12 11:54:40 +00:00
Patrick Robertson 70f155dfce add more of the USPs to the readme 2025-02-12 11:48:51 +00:00
Patrick Robertson 86254bdd4e Fix link in how to 2025-02-12 11:48:01 +00:00
Patrick Robertson 17f13db56c Make that code block a shell 2025-02-12 11:45:09 +00:00
Patrick Robertson d776be8a81 Fix links to docs 2025-02-12 11:41:54 +00:00
Patrick Robertson 460a71649c
Merge pull request #190 from bellingcat/docs_update 
Docs improvement
 2025-02-12 12:38:04 +01:00
Patrick Robertson a0c4a82825 Improved docstrings for base modules 2025-02-12 11:32:13 +00:00
Patrick Robertson 8054ea96b3 Document double dash between command line args and feeder urls on command line 2025-02-12 11:20:15 +00:00
Patrick Robertson de79e17128 better wording in link 2025-02-12 11:19:21 +00:00
Patrick Robertson d28d99daa6 Docs tidy ups and re-organising 2025-02-12 11:16:17 +00:00
Patrick Robertson 5b481f72ab Tidy ups to manifests for docs 2025-02-11 20:03:53 +00:00
Patrick Robertson 4c119b4db8 Add missing manifest for atlos_storage 2025-02-11 20:03:45 +00:00
Patrick Robertson d8f47ff9e4 Add multi-type modules to all headings on TOC 2025-02-11 19:46:57 +00:00
Patrick Robertson 1ee7981c6e Add YAML config to the module docs 2025-02-11 19:42:03 +00:00
Patrick Robertson 22fa9ba456 further tweaks 2025-02-11 18:52:28 +00:00
Patrick Robertson 1b976f4c09 Remove unused atlos util functions 2025-02-11 18:49:54 +00:00
Patrick Robertson 756f46012b Remove empty file 2025-02-11 18:47:54 +00:00
Patrick Robertson 1d69053dd5 Upgrade certifi 2025-02-11 18:41:37 +00:00
Patrick Robertson 40a95f7348 Add likify deps 2025-02-11 18:40:44 +00:00
Patrick Robertson a307d09e67 Readthedocs now requires all packages for running the pre-build scripts 2025-02-11 18:38:41 +00:00
Patrick Robertson 2c87474186 Change path for scripts 2025-02-11 18:37:03 +00:00
Patrick Robertson e65a99078f Slight tweaks of toctrees + reordering 2025-02-11 18:28:21 +00:00
Patrick Robertson e43dda2817
Merge pull request #185 from bellingcat/load_modules 
Refactor auto-archiver to use a modular structure for feeders/extractors/enrichers etc.
 2025-02-11 19:21:46 +01:00
Patrick Robertson 3787577a96 Screenshot enricher depends on geckodriver not chromedriver 2025-02-11 18:18:52 +00:00
Patrick Robertson ea728a7a97 TODO on facebook dropin not working 2025-02-11 15:56:12 +00:00
msramalho 91f1ebf7b3
fix temp for yandex new shortlink 2025-02-11 15:23:16 +00:00
msramalho c720541de2
merge conflicts 2025-02-11 15:22:06 +00:00
msramalho e507fc81d2
improves mimetype guessing, previously file.sub.something would not have an extension 2025-02-11 15:02:49 +00:00
msramalho 5478ed3860
bsky fix media fetching 2025-02-11 15:02:00 +00:00
msramalho 47d1dc9d47
typing warnings fixed 2025-02-11 15:01:37 +00:00
Patrick Robertson 62154ddfef Further tweaks and fixes 2025-02-11 14:37:29 +00:00
Patrick Robertson 29901da601 Merge branch 'load_modules' into docs_update 2025-02-11 14:10:56 +00:00
Patrick Robertson 895c843f04 Add a cheat sheet for configs and better folder structure for core modules 2025-02-11 14:06:53 +00:00
Patrick Robertson 2f51d3917a Further addition to docs: creating modules, configurations, installation 2025-02-11 13:49:30 +00:00
Erin Clark aa5ac18d6a
Merge pull request #189 from bellingcat/add_module_tests 
- Add module tests
- Fix some storage related bugs 
- Separate the modules setup() method and base module config_setup()
 2025-02-11 13:11:41 +00:00
Patrick Robertson 7d87b858d6 Merge branch 'load_modules' into docs_update 2025-02-11 13:09:38 +00:00
erinhmclark c8cd7ea63c Merge branch 'load_modules' into add_module_tests 
# Conflicts:
#	src/auto_archiver/modules/telethon_extractor/telethon_extractor.py
 2025-02-11 13:08:08 +00:00
msramalho 977618b4ce
doc: adds note about telethon vs telegram extractors 2025-02-11 13:04:59 +00:00
msramalho d90d3cec28
fix telethon_extractor setup 2025-02-11 13:03:18 +00:00
msramalho 977f06c37a
renames api_db property for clarity 2025-02-11 12:56:33 +00:00
msramalho 5c59029221
updates api_db for new API endpoint 2025-02-11 12:53:58 +00:00
msramalho 4eeb39477c
improves gsheetdb feedback on retrieve sheet failure 2025-02-11 12:53:46 +00:00
msramalho 6fdd5f0e66
fix cases of single : vs :: in entrypoint 2025-02-11 12:53:12 +00:00