auto-archiver/docs/source/modules/extractor.md

# Extractor Modules

Extractor modules are used to extract the content of a given URL. Typically, one extractor will work for one website or platform (e.g. a Telegram extractor or an Instagram), however, there are several wide-ranging extractors which work for a wide range of websites.

Extractors that are able to extract content from a wide range of websites include:
1. Generic Extractor: parses videos and images on sites using the powerful yt-dlp library.
2. Wayback Machine Extractor: sends pages to the Wayback machine for archiving, and stores the link.
3. WACZ Extractor: runs a web browser to 'browse' the URL and save a copy of the page in WACZ format. 

```{include} autogen/extractor.md
```

```{toctree}
:maxdepth: 1
:hidden:
:glob:
autogen/extractor/*
```
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			`# Extractor Modules`

			`Extractor modules are used to extract the content of a given URL. Typically, one extractor will work for one website or platform (e.g. a Telegram extractor or an Instagram), however, there are several wide-ranging extractors which work for a wide range of websites.`

			`Extractors that are able to extract content from a wide range of websites include:`
			`1. Generic Extractor: parses videos and images on sites using the powerful yt-dlp library.`
WIP: Docs tidyups+add howto on logging and authentication (Authentication is WIP) 2025-02-19 10:29:05 +00:00			`2. Wayback Machine Extractor: sends pages to the Wayback machine for archiving, and stores the link.`
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			`3. WACZ Extractor: runs a web browser to 'browse' the URL and save a copy of the page in WACZ format.`

			```{include} autogen/extractor.md
Add a cheat sheet for configs and better folder structure for core modules 2025-02-11 14:06:53 +00:00			```

			```{toctree}
Finish how to on authentication 2025-02-20 15:33:50 +00:00			`:maxdepth: 1`
Further tweaks and fixes 2025-02-11 14:37:29 +00:00			`:hidden:`
Add a cheat sheet for configs and better folder structure for core modules 2025-02-11 14:06:53 +00:00			`:glob:`
			`autogen/extractor/*`
Use a script to auto-generate documentation for the core modules from the manifest file 2025-02-10 22:51:04 +00:00			```