docs: wording overhaul, improve Caddy info, update music importing

MR !1385
environments/review-docs-fix-d-3ergtz/deployments/8580
Janek 2021-11-25 23:57:42 +00:00 zatwierdzone przez JuniorJPDJ
rodzic 87ff5b4da0
commit 4df883678b
2 zmienionych plików z 126 dodań i 90 usunięć

Wyświetl plik

@ -1,21 +1,21 @@
Importing music from the server Importing music from the server
=============================== ===============================
Funkwhale can import music files that are located on the server assuming Funkwhale can import music files saved on the server
they readable by the Funkwhale application. Your music files should contain at assuming they are readable by the Funkwhale application.
least an ``artist``, ``album`` and ``title`` tags, but we recommend you tag Your music files should contain at least
it extensively using a proper tool, such as Beets or Musicbrainz Picard. ``artist``, ``album`` and ``title`` tags,
but we recommend you tag extensively using a proper tool,
such as Beets or Musicbrainz Picard.
Funkwhale supports two different import modes: Funkwhale supports two different import modes:
- copy (the default): files are copied into Funkwhale's internal storage. This means importing a 1GB library will result in the same amount of space being used by Funkwhale. - copy(default): files are copied into Funkwhale's internal storage. This means importing a 1GB library will result in the same amount of space being used by Funkwhale.
- :ref:`in-place <in-place-import>` (when the ``--in-place`` is provided): files are referenced in Funkwhale's DB but not copied or touched in anyway. This is useful if you have a huge library, or one that is updated by an external tool such as Beets. - :ref:`in-place <in-place-import>` (with ``--in-place`` flag): files are referenced in Funkwhale's DB but not copied or touched in anyway. This is useful if you have a huge library, or one that is updated by an external tool such as Beets.
.. note:: Regardless of the mode you choose,
follow the below steps to import music,
In Funkwhale 1.0, **the default behaviour will change to in-place import** assuming your files are located in
Regardless of the mode you're choosing, import works as described below, assuming your files are located in
``/srv/funkwhale/data/music``: ``/srv/funkwhale/data/music``:
.. code-block:: bash .. code-block:: bash
@ -24,17 +24,18 @@ Regardless of the mode you're choosing, import works as described below, assumin
python api/manage.py import_files $LIBRARY_ID "/srv/funkwhale/data/music/" --recursive --noinput python api/manage.py import_files $LIBRARY_ID "/srv/funkwhale/data/music/" --recursive --noinput
.. note:: .. note::
You'll have to create a library in the Web UI before to get your library ID. Simply visit You have to create a library in the Web UI to get your library ID.
https://yourdomain/content/libraries/ to create one. Simply visit https://yourdomain/content/libraries/ to create one.
Library IDs are available in library urls or sharing link. In this example: Library IDs are part of the library url or sharing link.
For example, the library ID of
https://funkwhale.instance/content/libraries/769a2ae3-eb3d-4aff-9f94-2c4d80d5c2d1, https://funkwhale.instance/content/libraries/769a2ae3-eb3d-4aff-9f94-2c4d80d5c2d1,
the library ID is 769a2bc3-eb1d-4aff-9f84-2c4d80d5c2d1 is 769a2bc3-eb1d-4aff-9f84-2c4d80d5c2d1
You can use only the first characters of the ID when calling the command, like that: You can use only the first characters of the ID when calling the command, like that:
``export LIBRARY_ID="769a2bc3"`` ``export LIBRARY_ID="769a2bc3"``
When you use docker, the ``/srv/funkwhale/data/music`` is mounted from the host When you use docker, ``/srv/funkwhale/data/music`` is mounted from the host
to the ``/music`` directory on the container: to the ``/music`` directory on the container:
.. code-block:: bash .. code-block:: bash
@ -50,25 +51,24 @@ When you installed Funkwhale via ansible, you need to call a script instead of P
/srv/funkwhale/manage import_files $LIBRARY_ID "/srv/funkwhale/data/music/" --recursive --noinput /srv/funkwhale/manage import_files $LIBRARY_ID "/srv/funkwhale/data/music/" --recursive --noinput
The import command supports several options, and you can check the help to The import command supports several options,
get details:: check the help for details::
docker-compose run --rm api python manage.py import_files --help docker-compose run --rm api python manage.py import_files --help
.. note:: .. note::
For the best results, we recommend tagging your music collection through We recommend tagging your music collection using `Picard <http://picard.musicbrainz.org/>`_ to have the best quality metadata.
`Picard <http://picard.musicbrainz.org/>`_ in order to have the best quality metadata.
.. note:: .. note::
This command is idempotent, meaning you can run it multiple times on the same This command is idempotent,
files and already imported files will simply be skipped. meaning you can run it multiple times on the same files
and already imported files are simply skipped.
.. note:: .. note::
At the moment, only Flac, OGG/Vorbis and MP3 or AIFF files with ID3 tags are supported At the moment, only Flac, OGG/Vorbis and MP3 or AIFF files with ID3 tags are supported.
.. _in-place-import: .. _in-place-import:
@ -76,49 +76,60 @@ get details::
In-place import In-place import
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
By default, the CLI-importer will copy imported files to Funkwhale's internal By default, the CLI-importer will copy imported files to Funkwhale's internal storage.
storage. This means importing a 1GB library will result in the same amount This means importing a 1GB library will result
of space being used by Funkwhale. in the same amount of space being used by Funkwhale.
While this behaviour has some benefits (easier backups and configuration), While this behaviour has some benefits (easier backups and configuration),
it's not always the best choice, especially if you have a huge library it is not always the best choice,
to import and don't want to double your disk usage. especially if you have a huge library to import
and don't want to double your disk usage.
The CLI importer supports an additional ``--in-place`` option that triggers the The CLI importer supports an additional ``--in-place`` option
following behaviour during import: through which Funkwhale will store file paths rather than file content.
1. Imported files are not store in Funkwhale anymore Structure
2. Instead, Funkwhale will store the file path and use it to serve the music *********
Because those files are not managed by Funkwhale, we offer additional Because imported files are not managed by Funkwhale,
configuration options to ensure the webserver can serve them properly: we offer additional configuration options
to ensure the webserver can serve them properly:
- :data:`MUSIC_DIRECTORY_PATH <config.settings.common.MUSIC_DIRECTORY_PATH>` - :data:`MUSIC_DIRECTORY_PATH <config.settings.common.MUSIC_DIRECTORY_PATH>`
- :data:`MUSIC_DIRECTORY_SERVING_PATH <config.settings.common.MUSIC_DIRECTORY_SERVE_PATH>` - :data:`MUSIC_DIRECTORY_SERVING_PATH <config.settings.common.MUSIC_DIRECTORY_SERVE_PATH>`
We recommend you symlink all your music directories into ``/srv/funkwhale/data/music`` We recommend you symlink all your music directories into ``/srv/funkwhale/data/music``
and run the `import_files` command from that directory. This will make it possible and run the `import_files` command from that directory.
to use multiple music directories, without any additional configuration This will make it possible to use multiple music directories
on the webserver side. without any additional configuration on the webserver side.
For instance, if you have a NFS share with your music mounted at ``/media/mynfsshare``, For instance, if you have an NFS share
with your music mounted at ``/media/mynfsshare``,
you can create a symlink like this:: you can create a symlink like this::
ln -s /media/mynfsshare /srv/funkwhale/data/music/nfsshare ln -s /media/mynfsshare /srv/funkwhale/data/music/nfsshare
And import music from this share with this command:: And import music from the share::
export LIBRARY_ID="<your_libary_id>" export LIBRARY_ID="<your_libary_id>"
python api/manage.py import_files $LIBRARY_ID "/srv/funkwhale/data/music/nfsshare/" --recursive --noinput --in-place python api/manage.py import_files $LIBRARY_ID "/srv/funkwhale/data/music/nfsshare/" --recursive --noinput --in-place
On docker setups, it will require a bit more work, because while the ``/srv/funkwhale/data/music`` is mounted Docker
in containers, symlinked directories are not. ******
To fix that, you can use bind mounts instead of symbolic links, as it replicates the source directory tree. With the previous NFS share, it would go this way:: Docker setups require a bit more work,
because while the ``/srv/funkwhale/data/music`` is mounted in containers,
symlinked directories are not.
To fix that, you can use bind mounts instead of symbolic links,
as they replicate the source directory tree.
With the previous NFS share, use this command::
mount --bind /media/mynfsshare /srv/funkwhale/data/music/nfsshare mount --bind /media/mynfsshare /srv/funkwhale/data/music/nfsshare
If you want to go with symlinks, ensure each symlinked directory is mounted as a volume as well in your ``docker-compose.yml`` file:: If you want to go with symlinks,
ensure each symlinked directory is mounted as a volume
as well as in your ``docker-compose.yml`` file::
celeryworker: celeryworker:
volumes: volumes:
@ -137,15 +148,16 @@ If you want to go with symlinks, ensure each symlinked directory is mounted as a
Metadata updates Metadata updates
^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
When doing an import with in ``in-place`` mode, the importer will also check and update existing entries When doing an import with in ``in-place`` mode,
found in the database. For instance, if a file was imported, the ID3 Title tag was updated, and you rerun a scan, the importer will also check and update existing entries found in the database.
Funkwhale will pick up the new title. The following fields can be updated this way: For instance, if the ID3 Title tag of an existing song was updated since the last scan, Funkwhale picks up the new title.
The following fields can be updated this way:
- Track mbid - Track mbid
- Track title - Track title
- Track position and disc number - Track position and disc number
- Track license and copyright - Track license and copyright
- Track genre - Track genre (`from version 1.2 <https://dev.funkwhale.audio/funkwhale/funkwhale/-/merge_requests/1225>`_)
- Album cover - Album cover
- Album title - Album title
- Album mbid - Album mbid
@ -155,17 +167,23 @@ Funkwhale will pick up the new title. The following fields can be updated this w
- Album artist name - Album artist name
- Album artist mbid - Album artist mbid
Changes in artist name can lead to multiple artists with the same name in the database,
`this is a known issue <https://dev.funkwhale.audio/funkwhale/funkwhale/-/issues/1318>`_
and can be remedied by adding mbids.
React to filesystem events with ``--watch`` React to filesystem events with ``--watch``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you have a really big library or one that is updated quite often, running the ``import_files`` command by hand If you have a really big library or update it regularly,
may not be practical. To help with this use case, the ``import_files`` command supports a ``--watch`` flag that will observes filesystem events running the ``import_files`` command by hand may not be practical.
instead of performing a full import. For this use case,
the ``import_files`` command supports a ``--watch`` flag
through which it observes filesystem events instead of performing a full import.
File creation, move, update and removal are handled when ``--watch`` is provided: File creation, move, update and removal
are handled when ``--watch`` is provided:
- Files created in the watched directory are imported immediatly - Files created in the watched directory are imported immediately
- If using ``in-place`` mode, files updates trigger a metadata update on the corresponding entries - If using ``in-place`` mode, files updates trigger a metadata update on the corresponding entries
- If using ``in-place`` mode, files that are moved and known by Funkwhale will see their path updated in Funkwhale's DB - If using ``in-place`` mode, files that are moved and known by Funkwhale will see their path updated in Funkwhale's DB
- If using ``in-place`` mode, files that are removed and known by Funkwhale will be removed from Funkwhale's DB - If using ``in-place`` mode, files that are removed and known by Funkwhale will be removed from Funkwhale's DB
@ -173,7 +191,8 @@ File creation, move, update and removal are handled when ``--watch`` is provided
Pruning dangling metadata with ``--prune`` Pruning dangling metadata with ``--prune``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Funkwhale is, by design, conservative with music metadata in its database. If you remove a file from Funkwhale's DB, Funkwhale is, by design, conservative with music metadata in its database.
If you remove a file from Funkwhale's DB,
the corresponding artist, album and track object won't be deleted by default. the corresponding artist, album and track object won't be deleted by default.
If you want to prune dangling metadata from the database once the ``import_files`` command is over, simply add the ``--prune`` flag. If you want to prune dangling metadata from the database once the ``import_files`` command is over, simply add the ``--prune`` flag.
@ -182,18 +201,20 @@ This also works in with ``--watch``.
Album covers Album covers
^^^^^^^^^^^^ ^^^^^^^^^^^^
Whenever possible, Funkwhale will import album cover, with the following precedence: Whenever possible, Funkwhale obtains album covers for tracks,
with the following precedence:
1. It will use the cover embedded in the audio files themeselves, if any (Flac/MP3 only) 1. The cover embedded in the audio files themeselves, if any (Flac/MP3 only)
2. It will use a cover.jpg or a cover.png file from the imported track directory, if any 2. Use a cover.jpg or a cover.png file from the imported track directory, if any
3. It will fetch cover art from musicbrainz, assuming the file is tagged correctly 3. Fetch cover art from musicbrainz, assuming the file is tagged correctly
Getting demo tracks Getting demo tracks
^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^
If you do not have any music on your server but still want to test the import If you do not have any music on your server
process, you can call the following methods do download a few albums licenced but want to test the import process,
under creative commons (courtesy of Jamendo): you can call the following methods
to download a few albums licenced under creative commons (courtesy of Jamendo):
.. parsed-literal:: .. parsed-literal::
@ -202,4 +223,5 @@ under creative commons (courtesy of Jamendo):
chmod +x download-tracks.sh chmod +x download-tracks.sh
./download-tracks.sh music.txt ./download-tracks.sh music.txt
This will download a bunch of zip archives (one per album) under the ``data/music`` directory and unzip their content. This will download a bunch of zip archives (one per album)
under the ``data/music`` directory and unzip their content.

Wyświetl plik

@ -4,15 +4,17 @@ Installation
Requirements Requirements
------------ ------------
Regardless of your chosen installation method, the following requirements must be met in order to successfully deploy Funkwhale: Regardless of your chosen installation method,
the following is required to successfully deploy Funkwhale:
- **A dedicated domain or subdomain**: it is not possible to deploy Funkwhale on a subdirectory of an existing domain. - **A dedicated domain or subdomain**: it is not possible to deploy Funkwhale on a subdirectory of an existing domain.
- **Access to ports 80 and/or 443**: if you cannot serve the Funkwhale web app and API on these ports, federation will not work - **Access to ports 80 and/or 443**: if you cannot serve the Funkwhale web app and API on these ports, federation will not work
.. note:: .. note::
Because of the federated nature of Funkwhale, **it is strongly recommended not to change the Funkwhale domain after initial deployment**, as it is likely to break Because of the federated nature of Funkwhale,
your installation. **it is strongly recommended not to change the Funkwhale domain after initial deployment**,
as it is likely to break your installation.
Project architecture Project architecture
-------------------- --------------------
@ -34,8 +36,9 @@ The project relies on the following components and services to work:
Hardware requirements Hardware requirements
--------------------- ---------------------
Funkwhale is not especially CPU hungry. On a dockerized instance with 2 CPUs Funkwhale is not especially CPU hungry.
and a few active users, the memory footprint is around 500Mb:: On a dockerized instance with 2 CPUs and a few active users,
the memory footprint is around 500Mb::
CONTAINER MEM USAGE CONTAINER MEM USAGE
funkwhale_api_1 202 MiB funkwhale_api_1 202 MiB
@ -148,10 +151,11 @@ Serving only the frontend
.. note:: .. note::
You do not need to do this if you are deploying using Docker, as frontend files You do not need to do this if you are deploying using Docker,
are already included in the docker image. as frontend files are already included in the docker image.
You also do not need to do this if you are deploying manually on Debian or Arch, as this is covered by the corresponding documentation already. You also do not need to do this if you are deploying manually on Debian or Arch,
as this is covered by the corresponding documentation already.
Files for the web frontend are purely static and can simply be downloaded, unzipped and served from any webserver: Files for the web frontend are purely static and can simply be downloaded, unzipped and served from any webserver:
@ -167,20 +171,23 @@ Files for the web frontend are purely static and can simply be downloaded, unzip
Reverse proxy configuration Reverse proxy configuration
--------------------------- ---------------------------
In order to make Funkwhale accessible from outside your server and to play nicely with other applications on your machine, you should configure a reverse proxy. In order to make Funkwhale accessible from outside your server
and to play nicely with other applications on your machine,
you should configure a reverse proxy.
We offer sample configurations for Nginx, Apache2 and Caddy. We offer sample configurations for Nginx, Apache2 and Caddy.
.. note:: .. note::
You can freely adapt the proposed configuration to your own needs, as we cannot You can freely adapt the proposed configuration to your own needs,
cover every use case with a single template, especially when it's related as we cannot cover every use case with a single template,
to SSL configuration. especially when it's related to SSL configuration.
Nginx Nginx
^^^^^ ^^^^^
Ensure you have a recent version of nginx on your server. On Debian-like system, you would have to run the following: Ensure you have a recent version of nginx on your server.
On a Debian-based system use apt:
.. code-block:: shell .. code-block:: shell
@ -193,9 +200,11 @@ On Arch Linux and its derivatives:
sudo pacman -S nginx sudo pacman -S nginx
To avoid configuration errors at this level, we will generate an nginx configuration To avoid configuration errors at this level,
using your .env file. This will ensure your reverse-proxy configuration always we will generate an nginx configuration using your .env file.
match the application configuration and make upgrade/maintenance easier. This will ensure your reverse-proxy configuration
always matches the application configuration
and makes upgrade/maintenance easier.
.. note:: .. note::
The following commands need to be run as superuser. The following commands need to be run as superuser.
@ -297,7 +306,8 @@ Add the following to your ``.env`` file::
REVERSE_PROXY_TYPE=apache2 REVERSE_PROXY_TYPE=apache2
Then restart Funkwhale. This is needed to ensure Funkwhale provides proper headers for media file serving. Then restart Funkwhale.
This is needed to ensure Funkwhale provides proper headers for media file serving.
Then, download our sample virtualhost file: Then, download our sample virtualhost file:
@ -306,18 +316,23 @@ Then, download our sample virtualhost file:
curl -L -o /etc/apache2/sites-available/funkwhale.conf "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/|version|/deploy/apache.conf" curl -L -o /etc/apache2/sites-available/funkwhale.conf "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/|version|/deploy/apache.conf"
ln -s /etc/apache2/sites-available/funkwhale.conf /etc/apache2/sites-enabled/ ln -s /etc/apache2/sites-available/funkwhale.conf /etc/apache2/sites-enabled/
You can tweak the configuration file according to your setup, especially the Tweak the configuration file according to your setup,
TLS configuration. Otherwise, defaults should work if you followed the especially the TLS configuration.
installation guide. Otherwise, defaults should work if you followed the installation guide.
Check the configuration is valid with ``apache2ctl configtest``, and once you're Check the configuration with ``apache2ctl configtest``
done, load the new configuration with ``service apache2 restart``. and once you're done,
load the new configuration with ``service apache2 restart``.
Caddy Caddy
^^^^^ ^^^^^
If you're using Caddy as a reverse proxy in front of your docker containers (either mono or multi-container setup), We currently do not support a Caddy-only setup, but you can
you can use the following Caddyfile configuration: `help develop it <https://dev.funkwhale.audio/funkwhale/funkwhale/-/merge_requests/1384>`_!
To employ Caddy as a reverse proxy in front of your docker containers
(either mono- or multi-container setup),
use the following Caddyfile configuration:
Caddy v2:: Caddy v2::
@ -338,7 +353,9 @@ Caddy v1::
HTTPS configuration HTTPS configuration
^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^
After configuring the reverse proxy, you need a SSL certificate to enable HTTPS on your server. After configuring the reverse proxy,
you need a SSL certificate to enable HTTPS on your server
(unless you use Caddy, which handles them automatically).
The default reverse proxy configuration assumes you have those available at ``/etc/letsencrypt/live/${FUNKWHALE_HOSTNAME}/``, which The default reverse proxy configuration assumes you have those available at ``/etc/letsencrypt/live/${FUNKWHALE_HOSTNAME}/``, which
is the path used by `certbot <https://certbot.eff.org/docs/>`_ when generating certificates with Let's Encrypt. is the path used by `certbot <https://certbot.eff.org/docs/>`_ when generating certificates with Let's Encrypt.
@ -365,6 +382,3 @@ a certificate, as shown below. These instructions are provided by `certbot <http
sudo certbot --apache -d yourfunkwhale.domain sudo certbot --apache -d yourfunkwhale.domain
This creates a valid certificate and edit the nginx or Apache2 configuration to use the new certificate. The certificate will be automatically renewed when they expire. This creates a valid certificate and edit the nginx or Apache2 configuration to use the new certificate. The certificate will be automatically renewed when they expire.