kopia lustrzana https://github.com/bellingcat/auto-archiver
Start fleshing out the docs more - rearrange, separate out modules section, move files over to md (from rst)
Patrick Robertson
rodzic
63aba6ad39
commit
824728739a
138
README.md
138
README.md
|
|
@ -14,72 +14,15 @@ Read the [article about Auto Archiver on bellingcat.com](https://www.bellingcat.
|
|||
|
||||
Python tool to automatically archive social media posts, videos, and images from a Google Sheets, the console, and more. Uses different archivers depending on the platform, and can save content to local storage, S3 bucket (Digital Ocean Spaces, AWS, ...), and Google Drive. If using Google Sheets as the source for links, it will be updated with information about the archived content. It can be run manually or on an automated basis.
|
||||
|
||||
There are 3 ways to use the auto-archiver:
|
||||
1. (easiest installation) via docker
|
||||
2. (local python install) `pip install auto-archiver`
|
||||
3. (legacy/development) clone and manually install from repo (see legacy [tutorial video](https://youtu.be/VfAhcuV2tLQ))
|
||||
|
||||
But **you always need a configuration/orchestration file**, which is where you'll configure where/what/how to archive. Make sure you read [orchestration](#orchestration).
|
||||
## Installation
|
||||
|
||||
For full For instructions on how to install auto-archiver, view the [Installation Guide](docs/source/installation.md)
|
||||
|
||||
## How to install and run the auto-archiver
|
||||
Quick run using docker:
|
||||
|
||||
### Option 1 - docker
|
||||
`docker pull bellingcat/auto-archiver && docker run`
|
||||
|
||||
[](https://hub.docker.com/r/bellingcat/auto-archiver)
|
||||
|
||||
Docker works like a virtual machine running inside your computer, it isolates everything and makes installation simple. Since it is an isolated environment when you need to pass it your orchestration file or get downloaded media out of docker you will need to connect folders on your machine with folders inside docker with the `-v` volume flag.
|
||||
|
||||
|
||||
1. install [docker](https://docs.docker.com/get-docker/)
|
||||
2. pull the auto-archiver docker [image](https://hub.docker.com/r/bellingcat/auto-archiver) with `docker pull bellingcat/auto-archiver`
|
||||
3. run the docker image locally in a container: `docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml` breaking this command down:
|
||||
1. `docker run` tells docker to start a new container (an instance of the image)
|
||||
2. `--rm` makes sure this container is removed after execution (less garbage locally)
|
||||
3. `-v $PWD/secrets:/app/secrets` - your secrets folder
|
||||
1. `-v` is a volume flag which means a folder that you have on your computer will be connected to a folder inside the docker container
|
||||
2. `$PWD/secrets` points to a `secrets/` folder in your current working directory (where your console points to), we use this folder as a best practice to hold all the secrets/tokens/passwords/... you use
|
||||
3. `/app/secrets` points to the path the docker container where this image can be found
|
||||
4. `-v $PWD/local_archive:/app/local_archive` - (optional) if you use local_storage
|
||||
1. `-v` same as above, this is a volume instruction
|
||||
2. `$PWD/local_archive` is a folder `local_archive/` in case you want to archive locally and have the files accessible outside docker
|
||||
3. `/app/local_archive` is a folder inside docker that you can reference in your orchestration.yml file
|
||||
|
||||
### Option 2 - python package
|
||||
|
||||
<details><summary><code>Python package instructions</code></summary>
|
||||
|
||||
1. make sure you have python 3.10 or higher installed
|
||||
2. install the package with your preferred package manager: `pip/pipenv/conda install auto-archiver` or `poetry add auto-archiver`
|
||||
3. test it's installed with `auto-archiver --help`
|
||||
4. run it with your orchestration file and pass any flags you want in the command line `auto-archiver --config secrets/orchestration.yaml` if your orchestration file is inside a `secrets/`, which we advise
|
||||
|
||||
You will also need [ffmpeg](https://www.ffmpeg.org/), [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases), and optionally [fonts-noto](https://fonts.google.com/noto). Similar to the local installation.
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
### Option 3 - local installation
|
||||
This can also be used for development.
|
||||
|
||||
<details><summary><code>Legacy instructions, only use if docker/package is not an option</code></summary>
|
||||
|
||||
|
||||
Install the following locally:
|
||||
1. [ffmpeg](https://www.ffmpeg.org/) must also be installed locally for this tool to work.
|
||||
2. [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases) on a path folder like `/usr/local/bin`.
|
||||
3. [Poetry](https://python-poetry.org/docs/#installation) for dependency management and packaging.
|
||||
4. (optional) [fonts-noto](https://fonts.google.com/noto) to deal with multiple unicode characters during selenium/geckodriver's screenshots: `sudo apt install fonts-noto -y`.
|
||||
|
||||
Clone and run:
|
||||
1. `git clone https://github.com/bellingcat/auto-archiver`
|
||||
2. `poetry install`
|
||||
3. `poetry run python -m src.auto_archiver --config secrets/orchestration.yaml`
|
||||
|
||||
Note: Add the plugin [poetry-shell-plugin](https://github.com/python-poetry/poetry-plugin-shell) and run `poetry shell` to activate the virtual environment.
|
||||
This allows you to run the auto-archiver without the `poetry run` prefix.
|
||||
|
||||
</details><br/>
|
||||
|
||||
# Orchestration
|
||||
The archiver work is orchestrated by the following workflow (we call each a **step**):
|
||||
|
|
@ -260,76 +203,3 @@ The "archive location" link contains the path of the archived file, in local sto
|
|||
|
||||
|
||||
---
|
||||
## Development
|
||||
Use `python -m src.auto_archiver --config secrets/orchestration.yaml` to run from the local development environment.
|
||||
|
||||
### Testing
|
||||
|
||||
Tests are split using `pytest.mark` into 'core' and 'download' tests. Download tests will hit the network and make API calls (e.g. Twitter, Bluesky etc.) and should be run regularly to make sure that APIs have not changed.
|
||||
|
||||
Tests can be run as follows:
|
||||
```
|
||||
# run core tests
|
||||
pytest -ra -v -m "not download" # or poetry run pytest -ra -v -m "not download"
|
||||
# run download tests
|
||||
pytest -ra -v -m "download" # or poetry run pytest -ra -v -m "download"
|
||||
# run all tests
|
||||
pytest -ra -v # or poetry run pytest -ra -v
|
||||
```
|
||||
|
||||
#### Docker development
|
||||
working with docker locally:
|
||||
* `docker compose up` to build the first time and run a local image with the settings in `secrets/orchestration.yaml`
|
||||
* To modify/pass additional command line args, use `docker compose run auto-archiver --config secrets/orchestration.yaml [OTHER ARGUMENTS]`
|
||||
* To rebuild after code changes, just pass the `--build` flag, e.g. `docker compose up --build`
|
||||
|
||||
|
||||
manual release to docker hub
|
||||
* `docker image tag auto-archiver bellingcat/auto-archiver:latest`
|
||||
* `docker push bellingcat/auto-archiver`
|
||||
|
||||
|
||||
### Building the Docs
|
||||
|
||||
The documentation is built using [Sphinx](https://www.sphinx-doc.org/en/master/) and [AutoAPI](https://sphinx-autoapi.readthedocs.io/en/latest/) and hosted on ReadTheDocs.
|
||||
To build the documentation locally, run the following commands:
|
||||
|
||||
**Install required dependencies:**
|
||||
- Install the docs group of dependencies:
|
||||
```shell
|
||||
# only the docs dependencies
|
||||
poetry install --only docs
|
||||
|
||||
# or for all dependencies
|
||||
poetry install
|
||||
```
|
||||
- Either use [poetry-plugin-shell](https://github.com/python-poetry/poetry-plugin-shell) to activate the virtual environment: `poetry shell`
|
||||
- Or prepend the following commands with `poetry run`
|
||||
|
||||
**Create the documentation:**
|
||||
- Build the documentation:
|
||||
```
|
||||
# Using makefile (Linux/macOS):
|
||||
make -C docs html
|
||||
|
||||
# or using sphinx directly (Windows/Linux/macOS):
|
||||
sphinx-build -b html docs/source docs/_build/html
|
||||
```
|
||||
- If you make significant changes and want a fresh build run: `make -C docs clean` to remove the old build files.
|
||||
|
||||
**Viewing the documentation:**
|
||||
```shell
|
||||
# to open the documentation in your browser.
|
||||
open docs/_build/html/index.html
|
||||
|
||||
# or run autobuild to automatically update the documentation when you make changes
|
||||
sphinx-autobuild docs/source docs/_build/html
|
||||
```
|
||||
|
||||
|
||||
|
||||
#### RELEASE
|
||||
* update version in [version.py](src/auto_archiver/version.py)
|
||||
* go to github releases > new release > use `vx.y.z` for matching version notation
|
||||
* package is automatically updated in pypi
|
||||
* docker image is automatically pushed to dockerhup
|
||||
|
|
|
|||
|
|
@ -0,0 +1,34 @@
|
|||
API Reference
|
||||
=============
|
||||
|
||||
These pages are intended for developers of the `auto-archiver` package,
|
||||
and include documentation on the core classes and functions used by
|
||||
the auto-archiver
|
||||
|
||||
|
||||
Core Classes
|
||||
------------
|
||||
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
|
||||
{% for page in pages|selectattr("is_top_level_object") %}
|
||||
{% if page.name == 'core' %}
|
||||
{{ page.include_path }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
Util Functions
|
||||
--------------
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
|
||||
{% for page in pages|selectattr("is_top_level_object") %}
|
||||
{% if page.name == 'utils' %}
|
||||
{{ page.include_path }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
|
||||
|
|
@ -0,0 +1 @@
|
|||
{% extends "python/data.rst" %}
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
{% if obj.display %}
|
||||
{% if is_own_page %}
|
||||
{{ obj.id }}
|
||||
{{ "=" * obj.id | length }}
|
||||
|
||||
{% endif %}
|
||||
{% set visible_children = obj.children|selectattr("display")|list %}
|
||||
{% set own_page_children = visible_children|selectattr("type", "in", own_page_types)|list %}
|
||||
{% if is_own_page and own_page_children %}
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
{% for child in own_page_children %}
|
||||
{{ child.include_path }}
|
||||
{% endfor %}
|
||||
|
||||
{% endif %}
|
||||
.. py:{{ obj.type }}:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}{% if obj.args %}({{ obj.args }}){% endif %}
|
||||
|
||||
{% for (args, return_annotation) in obj.overloads %}
|
||||
{{ " " * (obj.type | length) }} {{ obj.short_name }}{% if args %}({{ args }}){% endif %}
|
||||
|
||||
{% endfor %}
|
||||
{% if obj.bases %}
|
||||
{% if "show-inheritance" in autoapi_options %}
|
||||
|
||||
Bases: {% for base in obj.bases %}{{ base|link_objs }}{% if not loop.last %}, {% endif %}{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% if "show-inheritance-diagram" in autoapi_options and obj.bases != ["object"] %}
|
||||
.. autoapi-inheritance-diagram:: {{ obj.obj["full_name"] }}
|
||||
:parts: 1
|
||||
{% if "private-members" in autoapi_options %}
|
||||
:private-bases:
|
||||
{% endif %}
|
||||
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{% if obj.docstring %}
|
||||
|
||||
{{ obj.docstring|indent(3) }}
|
||||
{% endif %}
|
||||
{% for obj_item in visible_children %}
|
||||
{% if obj_item.type not in own_page_types %}
|
||||
|
||||
{{ obj_item.render()|indent(3) }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% if is_own_page and own_page_children %}
|
||||
{% set visible_attributes = own_page_children|selectattr("type", "equalto", "attribute")|list %}
|
||||
{% if visible_attributes %}
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. autoapisummary::
|
||||
|
||||
{% for attribute in visible_attributes %}
|
||||
{{ attribute.id }}
|
||||
{% endfor %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set visible_exceptions = own_page_children|selectattr("type", "equalto", "exception")|list %}
|
||||
{% if visible_exceptions %}
|
||||
Exceptions
|
||||
----------
|
||||
|
||||
.. autoapisummary::
|
||||
|
||||
{% for exception in visible_exceptions %}
|
||||
{{ exception.id }}
|
||||
{% endfor %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set visible_classes = own_page_children|selectattr("type", "equalto", "class")|list %}
|
||||
{% if visible_classes %}
|
||||
Classes
|
||||
-------
|
||||
|
||||
.. autoapisummary::
|
||||
|
||||
{% for klass in visible_classes %}
|
||||
{{ klass.id }}
|
||||
{% endfor %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set visible_methods = own_page_children|selectattr("type", "equalto", "method")|list %}
|
||||
{% if visible_methods %}
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. autoapisummary::
|
||||
|
||||
{% for method in visible_methods %}
|
||||
{{ method.id }}
|
||||
{% endfor %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
{% if obj.display %}
|
||||
{% if is_own_page %}
|
||||
{{ obj.id }}
|
||||
{{ "=" * obj.id | length }}
|
||||
|
||||
{% endif %}
|
||||
.. py:{{ obj.type }}:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.name }}{% endif %}
|
||||
{% if obj.annotation is not none %}
|
||||
|
||||
:type: {% if obj.annotation %} {{ obj.annotation }}{% endif %}
|
||||
{% endif %}
|
||||
{% if obj.value is not none %}
|
||||
|
||||
{% if obj.value.splitlines()|count > 1 %}
|
||||
:value: Multiline-String
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<details><summary>Show Value</summary>
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
{{ obj.value|indent(width=6,blank=true) }}
|
||||
|
||||
.. raw:: html
|
||||
|
||||
</details>
|
||||
|
||||
{% else %}
|
||||
:value: {{ obj.value|truncate(100) }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
{% if obj.docstring %}
|
||||
|
||||
{{ obj.docstring|indent(3) }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{% extends "python/class.rst" %}
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
{% if obj.display %}
|
||||
{% if is_own_page %}
|
||||
{{ obj.id }}
|
||||
{{ "=" * obj.id | length }}
|
||||
|
||||
{% endif %}
|
||||
.. py:function:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ obj.args }}){% if obj.return_annotation is not none %} -> {{ obj.return_annotation }}{% endif %}
|
||||
{% for (args, return_annotation) in obj.overloads %}
|
||||
|
||||
{%+ if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ args }}){% if return_annotation is not none %} -> {{ return_annotation }}{% endif %}
|
||||
{% endfor %}
|
||||
{% for property in obj.properties %}
|
||||
|
||||
:{{ property }}:
|
||||
{% endfor %}
|
||||
|
||||
{% if obj.docstring %}
|
||||
|
||||
{{ obj.docstring|indent(3) }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
{% if obj.display %}
|
||||
{% if is_own_page %}
|
||||
{{ obj.id }}
|
||||
{{ "=" * obj.id | length }}
|
||||
|
||||
{% endif %}
|
||||
.. py:method:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ obj.args }}){% if obj.return_annotation is not none %} -> {{ obj.return_annotation }}{% endif %}
|
||||
{% for (args, return_annotation) in obj.overloads %}
|
||||
|
||||
{%+ if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ args }}){% if return_annotation is not none %} -> {{ return_annotation }}{% endif %}
|
||||
{% endfor %}
|
||||
{% for property in obj.properties %}
|
||||
|
||||
:{{ property }}:
|
||||
{% endfor %}
|
||||
|
||||
{% if obj.docstring %}
|
||||
|
||||
{{ obj.docstring|indent(3) }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
{% if obj.display %}
|
||||
{% if is_own_page %}
|
||||
{{ obj.id }}
|
||||
{{ "=" * obj.id|length }}
|
||||
|
||||
.. py:module:: {{ obj.name }}
|
||||
|
||||
{% if obj.docstring %}
|
||||
.. autoapi-nested-parse::
|
||||
|
||||
{{ obj.docstring|indent(3) }}
|
||||
|
||||
{% endif %}
|
||||
|
||||
{% block submodules %}
|
||||
{% set visible_subpackages = obj.subpackages|selectattr("display")|list %}
|
||||
{% set visible_submodules = obj.submodules|selectattr("display")|list %}
|
||||
{% set visible_submodules = (visible_subpackages + visible_submodules)|sort %}
|
||||
{% if visible_submodules %}
|
||||
Submodules
|
||||
----------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
{% for submodule in visible_submodules %}
|
||||
{{ submodule.include_path }}
|
||||
{% endfor %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
{% block content %}
|
||||
{% set visible_children = obj.children|selectattr("display")|list %}
|
||||
{% if visible_children %}
|
||||
{% set visible_attributes = visible_children|selectattr("type", "equalto", "data")|list %}
|
||||
{% if visible_attributes %}
|
||||
{% if "attribute" in own_page_types or "show-module-summary" in autoapi_options %}
|
||||
Attributes
|
||||
----------
|
||||
|
||||
{% if "attribute" in own_page_types %}
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
{% for attribute in visible_attributes %}
|
||||
{{ attribute.include_path }}
|
||||
{% endfor %}
|
||||
|
||||
{% endif %}
|
||||
.. autoapisummary::
|
||||
|
||||
{% for attribute in visible_attributes %}
|
||||
{{ attribute.id }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set visible_exceptions = visible_children|selectattr("type", "equalto", "exception")|list %}
|
||||
{% if visible_exceptions %}
|
||||
{% if "exception" in own_page_types or "show-module-summary" in autoapi_options %}
|
||||
Exceptions
|
||||
----------
|
||||
|
||||
{% if "exception" in own_page_types %}
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
{% for exception in visible_exceptions %}
|
||||
{{ exception.include_path }}
|
||||
{% endfor %}
|
||||
|
||||
{% endif %}
|
||||
.. autoapisummary::
|
||||
|
||||
{% for exception in visible_exceptions %}
|
||||
{{ exception.id }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set visible_classes = visible_children|selectattr("type", "equalto", "class")|list %}
|
||||
{% if visible_classes %}
|
||||
{% if "class" in own_page_types or "show-module-summary" in autoapi_options %}
|
||||
Classes
|
||||
-------
|
||||
|
||||
{% if "class" in own_page_types %}
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
{% for klass in visible_classes %}
|
||||
{{ klass.include_path }}
|
||||
{% endfor %}
|
||||
|
||||
{% endif %}
|
||||
.. autoapisummary::
|
||||
|
||||
{% for klass in visible_classes %}
|
||||
{{ klass.id }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set visible_functions = visible_children|selectattr("type", "equalto", "function")|list %}
|
||||
{% if visible_functions %}
|
||||
{% if "function" in own_page_types or "show-module-summary" in autoapi_options %}
|
||||
Functions
|
||||
---------
|
||||
|
||||
{% if "function" in own_page_types %}
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
{% for function in visible_functions %}
|
||||
{{ function.include_path }}
|
||||
{% endfor %}
|
||||
|
||||
{% endif %}
|
||||
.. autoapisummary::
|
||||
|
||||
{% for function in visible_functions %}
|
||||
{{ function.id }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
|
||||
{% endif %}
|
||||
{% set this_page_children = visible_children|rejectattr("type", "in", own_page_types)|list %}
|
||||
{% if this_page_children %}
|
||||
{{ obj.type|title }} Contents
|
||||
{{ "-" * obj.type|length }}---------
|
||||
|
||||
{% for obj_item in this_page_children %}
|
||||
{{ obj_item.render()|indent(0) }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
{% else %}
|
||||
.. py:module:: {{ obj.name }}
|
||||
|
||||
{% if obj.docstring %}
|
||||
.. autoapi-nested-parse::
|
||||
|
||||
{{ obj.docstring|indent(6) }}
|
||||
|
||||
{% endif %}
|
||||
{% for obj_item in visible_children %}
|
||||
{{ obj_item.render()|indent(3) }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{% extends "python/module.rst" %}
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
{% if obj.display %}
|
||||
{% if is_own_page %}
|
||||
{{ obj.id }}
|
||||
{{ "=" * obj.id | length }}
|
||||
|
||||
{% endif %}
|
||||
.. py:property:: {% if is_own_page %}{{ obj.id}}{% else %}{{ obj.short_name }}{% endif %}
|
||||
{% if obj.annotation %}
|
||||
|
||||
:type: {{ obj.annotation }}
|
||||
{% endif %}
|
||||
{% for property in obj.properties %}
|
||||
|
||||
:{{ property }}:
|
||||
{% endfor %}
|
||||
|
||||
{% if obj.docstring %}
|
||||
|
||||
{{ obj.docstring|indent(3) }}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
|
@ -17,6 +17,7 @@ extensions = [
|
|||
'sphinxcontrib.mermaid', # Mermaid diagrams
|
||||
"sphinx.ext.viewcode", # Source code links
|
||||
"sphinx.ext.napoleon", # Google-style and NumPy-style docstrings
|
||||
"sphinx.ext.autosectionlabel",
|
||||
# "sphinx.ext.autodoc", # Include custom docstrings
|
||||
# 'sphinx.ext.autosummary', # Summarize module/class/function docs
|
||||
]
|
||||
|
|
@ -27,12 +28,13 @@ exclude_patterns = []
|
|||
|
||||
# -- AutoAPI Configuration ---------------------------------------------------
|
||||
autoapi_type = 'python'
|
||||
autoapi_dirs = ["../../src"]
|
||||
autoapi_dirs = ["../../src/auto_archiver/core/", "../../src/auto_archiver/utils/", "../../src/auto_archiver/modules/"]
|
||||
autodoc_typehints = "signature" # Include type hints in the signature
|
||||
autoapi_ignore = [] # Ignore specific modules
|
||||
autoapi_keep_files = True # Option to retain intermediate JSON files for debugging
|
||||
autoapi_ignore = ["*/version.py", ] # Ignore specific modules
|
||||
autoapi_keep_files = False # Option to retain intermediate JSON files for debugging
|
||||
autoapi_add_toctree_entry = True # Include API docs in the TOC
|
||||
autoapi_template_dir = None # Use default templates
|
||||
autoapi_python_use_implicit_namespaces = True
|
||||
autoapi_template_dir = "../_templates/autoapi"
|
||||
autoapi_options = [
|
||||
"members",
|
||||
"undoc-members",
|
||||
|
|
@ -53,6 +55,8 @@ myst_enable_extensions = [
|
|||
"linkify", # Auto-detect links
|
||||
"substitution", # Text substitutions
|
||||
]
|
||||
myst_heading_anchors = 2
|
||||
|
||||
source_suffix = {
|
||||
".rst": "restructuredtext",
|
||||
".md": "markdown",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
Core Modules
|
||||
============
|
||||
|
||||
These pages are intended for developers of the `auto-archiver` package, and include documentation on the core classes and functions used by the auto-archiver
|
||||
|
||||
.. toctree::
|
||||
:titlesonly:
|
||||
|
||||
{% for page in pages|selectattr("is_top_level_object") %}
|
||||
{{ page.include_path }}
|
||||
{% endfor %}
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
|
||||
Developer Guidelines
|
||||
====================
|
||||
|
||||
This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.
|
||||
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
|
||||
# Developer Guidelines
|
||||
|
||||
This section of the documentation provides guidelines for developers who want to modify or contribute to the tool.
|
||||
|
||||
|
||||
## Developer Install
|
||||
|
||||
1. Clone the project using `git clone https://github.com/bellingcat/auto-archiver.git`
|
||||
2. Install poetry using `curl -sSL https://install.python-poetry.org | python3 -` ([other installation methods](https://python-poetry.org/docs/#installation))
|
||||
3. Install dependencies with `poetry install`
|
||||
|
||||
## Running
|
||||
4. Run the code with `poetry run auto-archiver [my args]`
|
||||
|
||||
```{note}
|
||||
Add the plugin [poetry-shell-plugin](https://github.com/python-poetry/poetry-plugin-shell) and run `poetry shell` to activate the virtual environment.
|
||||
This allows you to run the auto-archiver without the `poetry run` prefix.
|
||||
```
|
||||
|
||||
### Optional Development Packages
|
||||
|
||||
Install development packages (used for unit tests etc.) using:
|
||||
`poetry install -with dev`
|
||||
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
docker_development
|
||||
testing
|
||||
docs
|
||||
release
|
||||
```
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
## Docker development
|
||||
working with docker locally:
|
||||
* `docker compose up` to build the first time and run a local image with the settings in `secrets/orchestration.yaml`
|
||||
* To modify/pass additional command line args, use `docker compose run auto-archiver --config secrets/orchestration.yaml [OTHER ARGUMENTS]`
|
||||
* To rebuild after code changes, just pass the `--build` flag, e.g. `docker compose up --build`
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
|
||||
### Building the Docs
|
||||
|
||||
The documentation is built using [Sphinx](https://www.sphinx-doc.org/en/master/) and [AutoAPI](https://sphinx-autoapi.readthedocs.io/en/latest/) and hosted on ReadTheDocs.
|
||||
To build the documentation locally, run the following commands:
|
||||
|
||||
**Install required dependencies:**
|
||||
- Install the docs group of dependencies:
|
||||
```shell
|
||||
# only the docs dependencies
|
||||
poetry install --only docs
|
||||
|
||||
# or for all dependencies
|
||||
poetry install
|
||||
```
|
||||
- Either use [poetry-plugin-shell](https://github.com/python-poetry/poetry-plugin-shell) to activate the virtual environment: `poetry shell`
|
||||
- Or prepend the following commands with `poetry run`
|
||||
|
||||
**Create the documentation:**
|
||||
- Build the documentation:
|
||||
```
|
||||
# Using makefile (Linux/macOS):
|
||||
make -C docs html
|
||||
|
||||
# or using sphinx directly (Windows/Linux/macOS):
|
||||
sphinx-build -b html docs/source docs/_build/html
|
||||
```
|
||||
- If you make significant changes and want a fresh build run: `make -C docs clean` to remove the old build files.
|
||||
|
||||
**Viewing the documentation:**
|
||||
```shell
|
||||
# to open the documentation in your browser.
|
||||
open docs/_build/html/index.html
|
||||
|
||||
# or run autobuild to automatically update the documentation when you make changes
|
||||
sphinx-autobuild docs/source docs/_build/html
|
||||
```
|
||||
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
# Release Process
|
||||
|
||||
```{note} This is a work in progress.
|
||||
```
|
||||
|
||||
1. Update the version number in [version.py](src/auto_archiver/version.py)
|
||||
2. Go to github releases > new release > use `vx.y.z` for matching version notation
|
||||
1. package is automatically updated in pypi
|
||||
2. docker image is automatically pushed to dockerhup
|
||||
|
||||
|
||||
|
||||
manual release to docker hub
|
||||
* `docker image tag auto-archiver bellingcat/auto-archiver:latest`
|
||||
* `docker push bellingcat/auto-archiver`
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
### Testing
|
||||
|
||||
Tests are split using `pytest.mark` into 'core' and 'download' tests. Download tests will hit the network and make API calls (e.g. Twitter, Bluesky etc.) and should be run regularly to make sure that APIs have not changed.
|
||||
|
||||
Tests can be run as follows:
|
||||
```
|
||||
# run core tests
|
||||
pytest -ra -v -m "not download" # or poetry run pytest -ra -v -m "not download"
|
||||
# run download tests
|
||||
pytest -ra -v -m "download" # or poetry run pytest -ra -v -m "download"
|
||||
# run all tests
|
||||
pytest -ra -v # or poetry run pytest -ra -v
|
||||
```
|
||||
|
|
@ -1,26 +1,18 @@
|
|||
.. auto-archiver documentation master file, created by
|
||||
sphinx-quickstart on Sun Jan 12 20:35:50 2025.
|
||||
You can adapt this file completely to your liking, but it should at least
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Auto Archiver documentation
|
||||
===========================
|
||||
|
||||
.. note::
|
||||
|
||||
This is a work in progress.
|
||||
|
||||
|
||||
.. include:: ../../README.md
|
||||
:parser: myst
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
:maxdepth: 2
|
||||
:hidden:
|
||||
:caption: Contents:
|
||||
|
||||
Overview <self>
|
||||
user_guidelines
|
||||
developer_guidelines
|
||||
configurations
|
||||
installation/installation
|
||||
development/developer_guidelines
|
||||
autoapi/index
|
||||
core_modules
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,13 @@
|
|||
Configurations
|
||||
==============
|
||||
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
configurations
|
||||
```
|
||||
|
||||
This section of the documentation provides guidelines for configuring the tool.
|
||||
|
||||
File Reference
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# Installing Auto Archiver
|
||||
|
||||
|
||||
There are 3 main ways to use the auto-archiver:
|
||||
1. Easiest: [via docker](#installing-with-docker)
|
||||
2. Local Install: [using pip](#local-installing-with-pip)
|
||||
3. Developer Install: [see the developer guidelines](../development/developer_guidelines)
|
||||
|
||||
|
||||
But **you always need a configuration/orchestration file**, which is where you'll configure where/what/how to archive. Make sure you read [orchestration](#orchestration).
|
||||
|
||||
|
||||
## Installing with Docker
|
||||
|
||||
[](https://hub.docker.com/r/bellingcat/auto-archiver)
|
||||
|
||||
Docker works like a virtual machine running inside your computer, it isolates everything and makes installation simple. Since it is an isolated environment when you need to pass it your orchestration file or get downloaded media out of docker you will need to connect folders on your machine with folders inside docker with the `-v` volume flag.
|
||||
|
||||
|
||||
1. install [docker](https://docs.docker.com/get-docker/)
|
||||
2. pull the auto-archiver docker [image](https://hub.docker.com/r/bellingcat/auto-archiver) with `docker pull bellingcat/auto-archiver`
|
||||
3. run the docker image locally in a container: `docker run --rm -v $PWD/secrets:/app/secrets -v $PWD/local_archive:/app/local_archive bellingcat/auto-archiver --config secrets/orchestration.yaml` breaking this command down:
|
||||
1. `docker run` tells docker to start a new container (an instance of the image)
|
||||
2. `--rm` makes sure this container is removed after execution (less garbage locally)
|
||||
3. `-v $PWD/secrets:/app/secrets` - your secrets folder
|
||||
1. `-v` is a volume flag which means a folder that you have on your computer will be connected to a folder inside the docker container
|
||||
2. `$PWD/secrets` points to a `secrets/` folder in your current working directory (where your console points to), we use this folder as a best practice to hold all the secrets/tokens/passwords/... you use
|
||||
3. `/app/secrets` points to the path the docker container where this image can be found
|
||||
4. `-v $PWD/local_archive:/app/local_archive` - (optional) if you use local_storage
|
||||
1. `-v` same as above, this is a volume instruction
|
||||
2. `$PWD/local_archive` is a folder `local_archive/` in case you want to archive locally and have the files accessible outside docker
|
||||
3. `/app/local_archive` is a folder inside docker that you can reference in your orchestration.yml file
|
||||
|
||||
## Local installing with Pip
|
||||
|
||||
1. make sure you have python 3.10 or higher installed
|
||||
2. install the package with your preferred package manager: `pip/pipenv/conda install auto-archiver` or `poetry add auto-archiver`
|
||||
3. test it's installed with `auto-archiver --help`
|
||||
4. run it with your orchestration file and pass any flags you want in the command line `auto-archiver --config secrets/orchestration.yaml` if your orchestration file is inside a `secrets/`, which we advise
|
||||
|
||||
### Installing Local Requirements
|
||||
|
||||
If using the local installation method, you will also need to install the following dependencies locally:
|
||||
|
||||
1. [ffmpeg](https://www.ffmpeg.org/) must also be installed locally for this tool to work.
|
||||
2. [firefox](https://www.mozilla.org/en-US/firefox/new/) and [geckodriver](https://github.com/mozilla/geckodriver/releases) on a path folder like `/usr/local/bin`.
|
||||
3. (optional) [fonts-noto](https://fonts.google.com/noto) to deal with multiple unicode characters during selenium/geckodriver's screenshots: `sudo apt install fonts-noto -y`.
|
||||
|
||||
|
||||
|
||||
|
||||
## Developer Install
|
||||
|
||||
[See the developer guidelines](../development/developer_guidelines)
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
|
||||
# User Guidelines
|
||||
|
||||
|
||||
This section of the documentation provides guidelines for users who want to use the tool,
|
||||
without needing to modify the code.
|
||||
To see the developer guidelines, see the [](development/developer_guidelines)
|
||||
|
||||
```{note}
|
||||
This is a work in progress.
|
||||
```
|
||||
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
|
||||
User Guidelines
|
||||
===============
|
||||
|
||||
This section of the documentation provides guidelines for users who want to use the tool,
|
||||
without needing to modify the code.
|
||||
To see the developer guidelines, see :ref:`developer_guidelines`.
|
||||
|
||||
.. note::
|
||||
|
||||
This is a work in progress.
|
||||
Ładowanie…
Reference in New Issue