Wykres commitów

14 Commity (24fbbd0644ba8ee3e9d730bdce68c31008f60904)

Autor SHA1 Wiadomość Data
Patrick Robertson 17d2d14680 Fix running 'cleanup' method on extractors that fail to start 2025-03-26 22:52:52 +04:00
Patrick Robertson 6700250891 Add a test for checking module type on setup 2025-03-20 18:18:53 +04:00
erinhmclark 8673bc5979 Fix unused imports and include rule. 2025-03-13 13:55:31 +00:00
erinhmclark 753c3c6214 Linting tests 2025-03-12 14:27:45 +00:00
erinhmclark 85abe1837a Ruff format with defaults. 2025-03-10 18:44:54 +00:00
erinhmclark d775e4612e Update gsheet_feeder references in tests. 2025-03-04 14:06:46 +00:00
Patrick Robertson 4174285898 Fix unit tests 2025-02-20 13:18:06 +00:00
Patrick Robertson 40b8359348 Implementation test with 2 x orchestrators with different configs 2025-02-20 11:18:28 +00:00
Patrick Robertson a9802dd004 Remove the global _LAZY_LOADED_MODULES and allow each instance of ArchivingOrchestrator to load its own modules 2025-02-19 12:25:35 +00:00
Patrick Robertson 3c543a3a6a
Various fixes for issues with new architecture (#208)
* Add formatters to the TOC - fixes #204

* Add 'steps' settings to the example YAML in the docs. Fixes #206

* Improved docs on authentication architecture

* Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending

* Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there)

* Rename the command line entrypoint to _command_line_run

Also: make it clear that code implementation should not call this
Make sure the command line entry returns (we don't want a generator)

* Fix unit tests to use now code-entry points

* Version bump

* Move iterating of generator up to __main__

* Breakpoint

* two minor fixes

* Fix unit tests + add new '__main__' entry point implementation test

* Skip youtube tests if running on CI. Should still run them locally

* Fix full implementation run on GH actions

* Fix skipif test for GH Actions CI

* Add skipifs for truth - it blocks GH:

---------

Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com>
2025-02-18 19:10:09 +00:00
Patrick Robertson 6ab8fd2ee4 Tidy up setting modules as Orchestrator attributes on startup.
Don't override the values in config['steps'] – the config should be left as is
2025-02-06 10:20:05 +01:00
Patrick Robertson d6b4b7a932 Further cleanup
* Removes (partly) the ArchivingOrchestrator
* Removes the cli_feeder module, and makes it the 'default', allowing you to pass URLs directly on the command line, without having to use the cumbersome --cli_feeder.urls. Just do auto-archiver https://my.url.com
* More unit tests
* Improved error handling
2025-01-30 16:44:40 +01:00
Patrick Robertson fade68c6f4 Fix up unit tests - dataclass + subclasses not having @dataclass was breaking it 2025-01-30 13:45:24 +01:00
Patrick Robertson b7d9145f6c Further tidyups + refactoring for new structure
* Add implementation tests for orchestrator + logging tests
* Standardise method/class vars for extractors to see if they are suitable
* Fix bugs with removing default loguru logger (allows further customisation)
* Fix bug loading required fields from file
*
2025-01-30 13:21:10 +01:00