auto-archiver/docs/source/core_modules.md

# Module Documentation

These pages describe the core modules that come with Auto Archiver and provide the main functionality for archiving websites on the internet. There are five core module types:

1. Feeders - these 'feed' information (the URLs) from various sources to the Auto Archiver for processing
2. Extractors - these 'extract' the page data for a given URL that is fed in by a feeder
3. Enrichers - these 'enrich' the data extracted in the previous step with additional information
4. Storage - these 'store' the data in a persistent location (on disk, Google Drive etc.)
5. Databases - these 'store' the status of the entire archiving process in a log file or database.


```{include} modules/autogen/module_list.md
```


```{toctree}
:maxdepth: 1
:caption: Core Modules
:hidden:

modules/config_cheatsheet
modules/feeder
modules/extractor
modules/enricher
modules/storage
modules/database
modules/formatter
```
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			`# Module Documentation`

Use 'Auto Archiver' naming for consistency. auto-archiver is reserved in the docs for when talking about the command line usage 2025-02-20 11:50:29 +00:00			`These pages describe the core modules that come with Auto Archiver and provide the main functionality for archiving websites on the internet. There are five core module types:`
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00
Use 'Auto Archiver' naming for consistency. auto-archiver is reserved in the docs for when talking about the command line usage 2025-02-20 11:50:29 +00:00			`1. Feeders - these 'feed' information (the URLs) from various sources to the Auto Archiver for processing`
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			`2. Extractors - these 'extract' the page data for a given URL that is fed in by a feeder`
			`3. Enrichers - these 'enrich' the data extracted in the previous step with additional information`
			`4. Storage - these 'store' the data in a persistent location (on disk, Google Drive etc.)`
			`5. Databases - these 'store' the status of the entire archiving process in a log file or database.`


			```{include} modules/autogen/module_list.md
			```


			```{toctree}
			`:maxdepth: 1`
			`:caption: Core Modules`
			`:hidden:`

Add a cheat sheet for configs and better folder structure for core modules 2025-02-11 14:06:53 +00:00			`modules/config_cheatsheet`
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			`modules/feeder`
			`modules/extractor`
			`modules/enricher`
			`modules/storage`
			`modules/database`
Various fixes for issues with new architecture (#208) * Add formatters to the TOC - fixes #204 * Add 'steps' settings to the example YAML in the docs. Fixes #206 * Improved docs on authentication architecture * Fix setting modules on the command line - they now override any module settings in the orchestration as opposed to appending * Fix tests for gsheet-feeder: add a test service_account.json (note: not real keys in there) * Rename the command line entrypoint to _command_line_run Also: make it clear that code implementation should not call this Make sure the command line entry returns (we don't want a generator) * Fix unit tests to use now code-entry points * Version bump * Move iterating of generator up to __main__ * Breakpoint * two minor fixes * Fix unit tests + add new '__main__' entry point implementation test * Skip youtube tests if running on CI. Should still run them locally * Fix full implementation run on GH actions * Fix skipif test for GH Actions CI * Add skipifs for truth - it blocks GH: --------- Co-authored-by: msramalho <19508417+msramalho@users.noreply.github.com> 2025-02-18 19:10:09 +00:00			`modules/formatter`
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			```