diff --git a/CONTRIBUTING b/CONTRIBUTING new file mode 100644 index 000000000..9f4ec8850 --- /dev/null +++ b/CONTRIBUTING @@ -0,0 +1,442 @@ +Contibute to Funkwhale development +================================== + +First of all, thank you for your interest in the project! We really +appreciate the fact that you're about to take some time to read this +and hack on the project. + +This document will guide you through common operations such as: + +- Setup your development environment +- Working on your first issue +- Writing unit tests to validate your work +- Submit your work + + +Setup your development environment +---------------------------------- + +If you want to fix a bug or implement a feature, you'll need +to run a local, development copy of funkwhale. + +We provide a docker based development environment, which should +be both easy to setup and work similarly regardless of your +development machine setup. + +Instructions for bare-metal setup will come in the future (Merge requests +are welcome). + +Installing docker and docker-compose +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This is already cover in the relevant documentations: + +- https://docs.docker.com/install/ +- https://docs.docker.com/compose/install/ + +Cloning the project +^^^^^^^^^^^^^^^^^^^ + +Visit https://code.eliotberriot.com/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Exemple using SSH:: + + git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git + cd funkwhale + + +A note about branches +^^^^^^^^^^^^^^^^^^^^^ + +Next release development occurs on the "develop" branch, and releases are made on the "master" branch. Therefor, when submitting Merge Requests, ensure you are merging on the develop branch. + + +Working with docker +^^^^^^^^^^^^^^^^^^^ + +In developpement, we use the docker-compose file named ``dev.yml``, and this is why all our docker-compose commands will look like this:: + + docker-compose -f dev.yml logs + +If you do not want to add the ``-f dev.yml`` snippet everytime, you can run this command before starting your work:: + + export COMPOSE_FILE=dev.yml + + +Building the containers +^^^^^^^^^^^^^^^^^^^^^^^ + +On your initial clone, or if there have been some changes in the +app dependencies, you will have to rebuild your containers. This is done +via the following command:: + + docker-compose -f dev.yml build + + +Creating your env file +^^^^^^^^^^^^^^^^^^^^^^ + +We provide a working .env.dev configuration file that is suitable for +development. However, to enable customization on your machine, you should +also create a .env file that will hold your personal environment +variables (those will not be commited to the project). + +Create it like this:: + + touch .env + + +Database management +^^^^^^^^^^^^^^^^^^^ + +To setup funkwhale's database schema, run this:: + + docker-compose -f dev.yml run --rm api python manage.py migrate + +This will create all the tables needed for the API to run proprely. +You will also need to run this whenever changes are made on the database +schema. + +It is safe to run this command multiple times, so you can run it whenever +you fetch develop. + + +Development data +^^^^^^^^^^^^^^^^ + +You'll need at least an admin user and some artists/tracks/albums to work +locally. + +Create an admin user with the following command:: + + docker-compose -f dev.yml run --rm api python manage.py createsuperuser + +Injecting fake data is done by running the fllowing script:: + + artists=25 + command="from funkwhale_api.music import fake_data; fake_data.create_data($artists)" + echo $command | docker-compose -f dev.yml run --rm api python manage.py shell -i python + +The previous command will create 25 artists with random albums, tracks +and metadata. + + +Launch all services +^^^^^^^^^^^^^^^^^^^ + +Then you can run everything with:: + + docker-compose -f dev.yml up + +This will launch all services, and output the logs in your current terminal window. +If you prefer to launch them in the background instead, use the ``-d`` flag, and access the logs when you need it via ``docker-compose -f dev.yml logs --tail=50 --follow``. + +Once everything is up, you can access the various funkwhale's components: + +- The Vue webapp, on http://localhost:8080 +- The API, on http://localhost:8080/api/v1/ +- The django admin, on http://localhost:8080/api/admin/ + +Stopping everything +^^^^^^^^^^^^^^^^^^^ + +Once you're down with your work, you can stop running containers, if any, with:: + + docker-compose -f dev.yml stop + + +Removing everything +^^^^^^^^^^^^^^^^^^^ + +If you want to wipe your development environment completely (e.g. if you want to start over from scratch), just run:: + + docker-compose -f dev.yml down -v + +This will wipe your containers and data, so please be careful before running it. + +You can keep your data by removing the ``-v`` flag. + + +Working with federation locally +------------------------------- + +This is not needed unless you need to work on federation-related features. + +To achieve that, you'll need: + +1. to update your dns resolver to resolve all your .dev hostnames locally +2. a reverse proxy (such as traefik) to catch those .dev requests and + and with https certificate +3. two instances (or more) running locally, following the regular dev setup + +Resolve .dev names locally +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you use dnsmasq, this is as simple as doing:: + + echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf + sudo systemctl restart dnsmasq + +If you use NetworkManager with dnsmasq integration, use this instead:: + + echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf + sudo systemctl restart NetworkManager + +Add wildcard certificate to the trusted certificates +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Simply copy bundled certificates:: + + sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/ + sudo update-ca-certificates + +This certificate is a wildcard for ``*.funkwhale.test`` + +Run a reverse proxy for your instances +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Create docker network +^^^^^^^^^^^^^^^^^^^^ + +Create the federation network:: + + docker network create federation + +Launch everything +^^^^^^^^^^^^^^^^^ + +Launch the traefik proxy:: + + docker-compose -f docker/traefik.yml up -d + +Then, in separate terminals, you can setup as many different instances as you +need:: + + export COMPOSE_PROJECT_NAME=node2 + docker-compose -f dev.yml run --rm api python manage.py migrate + docker-compose -f dev.yml run --rm api python manage.py createsuperuser + docker-compose -f dev.yml up nginx api front nginx api celeryworker + +Note that by default, if you don't export the COMPOSE_PROJECT_NAME, +we will default to node1 as the name of your instance. + +Assuming your project name is ``node1``, your server will be reachable +at ``https://node1.funkwhale.test/``. Not that you'll have to trust +the SSL Certificate as it's self signed. + +When working on federation with traefik, ensure you have this in your ``env``:: + + # This will ensure we don't bind any port on the host, and thus enable + # multiple instances of funkwhale to be spawned concurrently. + WEBPACK_DEVSERVER_PORT_BINDING= + # This disable certificate verification + EXTERNAL_REQUESTS_VERIFY_SSL=false + # this ensure you don't have incorrect urls pointing to http resources + FUNKWHALE_PROTOCOL=https + + +Typical workflow for a contribution +----------------------------------- + +0. Fork the project if you did not already or if you do not have access to the main repository +1. Checkout the development branch and pull most recent changes: ``git checkout develop && git pull`` +2. If working on an issue, assign yourself to the issue. Otherwise, consider open an issue before starting to work on something, especially for new features. +3. Create a dedicated branch for your work ``42-awesome-fix``. It is good practice to prefix your branch name with the ID of the issue you are solving. +4. Work on your stuff +5. Commit small, atomic changes to make it easier to review your contribution +6. Add a changelog fragment to summarize your changes: ``echo "Implemented awesome stuff (#42)" > changes/changelog.d/42.feature"`` +7. Push your branch +8. Create your merge request +9. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute! + + +Internationalization +-------------------- + +When working on the front-end, any end-user string should be translated +using either ```` or the ``$t('yourstring')`` +function. + +Extraction is done by calling ``yarn run i18n-extract``, which +will pull all the strings from source files and put them in a PO file. + +Contributing to the API +----------------------- + +Project structure +^^^^^^^^^^^^^^^^^ + +.. code-block:: shell + + tree api -L 2 -d + api + ├── config # configuration directory (settings, urls, wsgi server) + │ └── settings # Django settings files + ├── funkwhale_api # project directory, all funkwhale logic is here + ├── requirements # python requirements files + └── tests # test files, matches the structure of the funkwhale_api directory + +.. note:: + + Unless trivial, API contributions must include unittests to ensure + your fix or feature is working as expected and won't break in the future + +Running tests +^^^^^^^^^^^^^ + +To run the pytest test suite, use the following command:: + + docker-compose -f dev.yml run --rm api pytest + +This is regular pytest, so you can use any arguments/options that pytest usually accept:: + + # get some help + docker-compose -f dev.yml run --rm api pytest -h + # Stop on first failure + docker-compose -f dev.yml run --rm api pytest -x + # Run a specific test file + docker-compose -f dev.yml run --rm api pytest tests/test_acoustid.py + +Writing tests +^^^^^^^^^^^^^ + +Although teaching you how to write unit tests is outside of the scope of this +document, you'll find below a collection of tips, snippets and resources +you can use if you want to learn on that subject. + +Useful links: + +- `A quick introduction to unit test writing with pytest `_ +- `A complete guide to Test-Driven Development (although not using Pytest) `_ +- `pytest `_: documentation of our testing engine and runner +- `pytest-mock `_: project page of our mocking engine +- `factory-boy `_: documentation of factory-boy, which we use to easily generate fake objects and data + +Recommendations: + +- Test files must target a module and mimic ``funkwhale_api`` directory structure: if you're writing tests for ``funkwhale_api/myapp/views.py``, you should put thoses tests in ``tests/myapp/test_views.py`` +- Tests should be small and test one thing. If you need to test multiple things, write multiple tests. + +We provide a lot of utils and fixtures to make the process of writing tests as +painless as possible. You'll find some usage examples below. + +Use factories to create arbitrary objects: + +.. code-block:: python + + # funkwhale_api/myapp/users.py + + def downgrade_user(user): + """ + A simple function that remove superuser status from users + and return True if user was actually downgraded + """ + downgraded = user.is_superuser + user.is_superuser = False + user.save() + return downgraded + + # tests/myapp/test_users.py + from funkwhale_api.myapp import users + + def test_downgrade_superuser(factories): + user = factories['users.User'](is_superuser=True) + downgraded = users.downgrade_user(user) + + assert downgraded is True + assert user.is_superuser is False + + def test_downgrade_normal_user_does_nothing(factories): + user = factories['users.User'](is_superuser=False) + downgraded = something.downgrade_user(user) + + assert downgraded is False + assert user.is_superuser is False + +.. note:: + + We offer factories for almost if not all models. Factories are located + in a ``factories.py`` file inside each app. + +Mocking: mocking is the process of faking some logic in our code. This is +useful when testing components that depend on each other: + +.. code-block:: python + + # funkwhale_api/myapp/notifications.py + + def notify(email, message): + """ + A function that sends an email to the given recipient + with the given message + """ + + # our email sending logic here + # ... + + # funkwhale_api/myapp/users.py + from . import notifications + + def downgrade_user(user): + """ + A simple function that remove superuser status from users + and return True if user was actually downgraded + """ + downgraded = user.is_superuser + user.is_superuser = False + user.save() + if downgraded: + notifications.notify(user.email, 'You have been downgraded!') + return downgraded + + # tests/myapp/test_users.py + def test_downgrade_superuser_sends_email(factories, mocker): + """ + Your downgrade logic is already tested, however, we want to ensure + an email is sent when user is downgraded, but we don't have any email + server available in our testing environment. Thus, we need to mock + the email sending process. + """ + mocked_notify = mocker.patch('funkwhale_api.myapp.notifications.notify') + user = factories['users.User'](is_superuser=True) + users.downgrade_user(user) + + # here, we ensure our notify function was called with proper arguments + mocked_notify.assert_called_once_with(user.email, 'You have been downgraded') + + + def test_downgrade_not_superuser_skips_email(factories, mocker): + mocked_notify = mocker.patch('funkwhale_api.myapp.notifications.notify') + user = factories['users.User'](is_superuser=True) + users.downgrade_user(user) + + # here, we ensure no email was sent + mocked_notify.assert_not_called() + +Views: you can find some readable views tests in :file:`tests/users/test_views.py` + +.. note:: + + A complete list of available-fixtures is available by running + ``docker-compose -f dev.yml run --rm api pytest --fixtures`` + + +Contributing to the front-end +----------------------------- + +Running tests +^^^^^^^^^^^^^ + +To run the front-end test suite, use the following command:: + + docker-compose -f dev.yml run --rm front yarn run unit + +We also support a "watch and test" mode were we continually relaunch +tests when changes are recorded on the file system:: + + docker-compose -f dev.yml run --rm front yarn run unit-watch + +The latter is especially useful when you are debugging failing tests. + +.. note:: + + The front-end test suite coverage is still pretty low diff --git a/README.rst b/README.rst index 8a0ea4932..c8a126451 100644 --- a/README.rst +++ b/README.rst @@ -15,282 +15,8 @@ We offer various Matrix.org rooms to discuss about funkwhale: Please join those rooms if you have any questions! -Running the development version -------------------------------- +Contribute +---------- -If you want to fix a bug or implement a feature, you'll need -to run a local, development copy of funkwhale. - -We provide a docker based development environment, which should -be both easy to setup and work similarly regardless of your -development machine setup. - -Instructions for bare-metal setup will come in the future (Merge requests -are welcome). - -Installing docker and docker-compose -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This is already cover in the relevant documentations: - -- https://docs.docker.com/install/ -- https://docs.docker.com/compose/install/ - -Cloning the project -^^^^^^^^^^^^^^^^^^^ - -Visit https://code.eliotberriot.com/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Exemple using SSH:: - - git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git - cd funkwhale - - -A note about branches -^^^^^^^^^^^^^^^^^^^^^ - -Next release development occurs on the "develop" branch, and releases are made on the "master" branch. Therefor, when submitting Merge Requests, ensure you are merging on the develop branch. - - -Working with docker -^^^^^^^^^^^^^^^^^^^ - -In developpement, we use the docker-compose file named ``dev.yml``, and this is why all our docker-compose commands will look like this:: - - docker-compose -f dev.yml logs - -If you do not want to add the ``-f dev.yml`` snippet everytime, you can run this command before starting your work:: - - export COMPOSE_FILE=dev.yml - - -Building the containers -^^^^^^^^^^^^^^^^^^^^^^^ - -On your initial clone, or if there have been some changes in the -app dependencies, you will have to rebuild your containers. This is done -via the following command:: - - docker-compose -f dev.yml build - - -Creating your env file -^^^^^^^^^^^^^^^^^^^^^^ - -We provide a working .env.dev configuration file that is suitable for -development. However, to enable customization on your machine, you should -also create a .env file that will hold your personal environment -variables (those will not be commited to the project). - -Create it like this:: - - touch .env - - -Database management -^^^^^^^^^^^^^^^^^^^ - -To setup funkwhale's database schema, run this:: - - docker-compose -f dev.yml run --rm api python manage.py migrate - -This will create all the tables needed for the API to run proprely. -You will also need to run this whenever changes are made on the database -schema. - -It is safe to run this command multiple times, so you can run it whenever -you fetch develop. - - -Development data -^^^^^^^^^^^^^^^^ - -You'll need at least an admin user and some artists/tracks/albums to work -locally. - -Create an admin user with the following command:: - - docker-compose -f dev.yml run --rm api python manage.py createsuperuser - -Injecting fake data is done by running the fllowing script:: - - artists=25 - command="from funkwhale_api.music import fake_data; fake_data.create_data($artists)" - echo $command | docker-compose -f dev.yml run --rm api python manage.py shell -i python - -The previous command will create 25 artists with random albums, tracks -and metadata. - - -Launch all services -^^^^^^^^^^^^^^^^^^^ - -Then you can run everything with:: - - docker-compose -f dev.yml up - -This will launch all services, and output the logs in your current terminal window. -If you prefer to launch them in the background instead, use the ``-d`` flag, and access the logs when you need it via ``docker-compose -f dev.yml logs --tail=50 --follow``. - -Once everything is up, you can access the various funkwhale's components: - -- The Vue webapp, on http://localhost:8080 -- The API, on http://localhost:8080/api/v1/ -- The django admin, on http://localhost:8080/api/admin/ - - -Running API tests -^^^^^^^^^^^^^^^^^ - -To run the pytest test suite, use the following command:: - - docker-compose -f dev.yml run --rm api pytest - -This is regular pytest, so you can use any arguments/options that pytest usually accept:: - - # get some help - docker-compose -f dev.yml run --rm api pytest -h - # Stop on first failure - docker-compose -f dev.yml run --rm api pytest -x - # Run a specific test file - docker-compose -f dev.yml run --rm api pytest tests/test_acoustid.py - - -Running front-end tests -^^^^^^^^^^^^^^^^^^^^^^^ - -To run the front-end test suite, use the following command:: - - docker-compose -f dev.yml run --rm front yarn run unit - -We also support a "watch and test" mode were we continually relaunch -tests when changes are recorded on the file system:: - - docker-compose -f dev.yml run --rm front yarn run unit-watch - -The latter is especially useful when you are debugging failing tests. - -.. note:: - - The front-end test suite coverage is still pretty low - - -Stopping everything -^^^^^^^^^^^^^^^^^^^ - -Once you're down with your work, you can stop running containers, if any, with:: - - docker-compose -f dev.yml stop - - -Removing everything -^^^^^^^^^^^^^^^^^^^ - -If you want to wipe your development environment completely (e.g. if you want to start over from scratch), just run:: - - docker-compose -f dev.yml down -v - -This will wipe your containers and data, so please be careful before running it. - -You can keep your data by removing the ``-v`` flag. - - -Typical workflow for a merge request -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -0. Fork the project if you did not already or if you do not have access to the main repository -1. Checkout the development branch and pull most recent changes: ``git checkout develop && git pull`` -2. Create a dedicated branch for your work ``42-awesome-fix``. It is good practice to prefix your branch name with the ID of the issue you are solving. -3. Work on your stuff -4. Commit small, atomic changes to make it easier to review your contribution -5. Add a changelog fragment to summarize your changes: ``echo "Implemented awesome stuff (#42)" > changes/changelog.d/42.feature"`` -6. Push your branch -7. Create your merge request -8. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute! - - -Internationalization --------------------- - -When working on the front-end, any end-user string should be translated -using either ```` or the ``$t('yourstring')`` -function. - -Extraction is done by calling ``yarn run i18n-extract``, which -will pull all the strings from source files and put them in a PO file. - - -Working with federation locally -------------------------------- - -To achieve that, you'll need: - -1. to update your dns resolver to resolve all your .dev hostnames locally -2. a reverse proxy (such as traefik) to catch those .dev requests and - and with https certificate -3. two instances (or more) running locally, following the regular dev setup - -Resolve .dev names locally -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you use dnsmasq, this is as simple as doing:: - - echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf - sudo systemctl restart dnsmasq - -If you use NetworkManager with dnsmasq integration, use this instead:: - - echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf - sudo systemctl restart NetworkManager - -Add wildcard certificate to the trusted certificates -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Simply copy bundled certificates:: - - sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/ - sudo update-ca-certificates - -This certificate is a wildcard for ``*.funkwhale.test`` - -Run a reverse proxy for your instances -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - -Create docker network -^^^^^^^^^^^^^^^^^^^^ - -Create the federation network:: - - docker network create federation - -Launch everything -^^^^^^^^^^^^^^^^^ - -Launch the traefik proxy:: - - docker-compose -f docker/traefik.yml up -d - -Then, in separate terminals, you can setup as many different instances as you -need:: - - export COMPOSE_PROJECT_NAME=node2 - docker-compose -f dev.yml run --rm api python manage.py migrate - docker-compose -f dev.yml run --rm api python manage.py createsuperuser - docker-compose -f dev.yml up nginx api front nginx api celeryworker - -Note that by default, if you don't export the COMPOSE_PROJECT_NAME, -we will default to node1 as the name of your instance. - -Assuming your project name is ``node1``, your server will be reachable -at ``https://node1.funkwhale.test/``. Not that you'll have to trust -the SSL Certificate as it's self signed. - -When working on federation with traefik, ensure you have this in your ``env``:: - - # This will ensure we don't bind any port on the host, and thus enable - # multiple instances of funkwhale to be spawned concurrently. - WEBPACK_DEVSERVER_PORT_BINDING= - # This disable certificate verification - EXTERNAL_REQUESTS_VERIFY_SSL=false - # this ensure you don't have incorrect urls pointing to http resources - FUNKWHALE_PROTOCOL=https +Contribution guidelines as well as development installation instructions +are outlined in `CONTRIBUTING `_ diff --git a/docs/contributing.rst b/docs/contributing.rst new file mode 100644 index 000000000..0e5a08ecf --- /dev/null +++ b/docs/contributing.rst @@ -0,0 +1 @@ +.. include:: ../CONTRIBUTING diff --git a/docs/index.rst b/docs/index.rst index e4f0e4422..481690b70 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -19,6 +19,7 @@ Funkwhale is a self-hosted, modern free and open-source music server, heavily in api upgrading third-party + contributing changelog Indices and tables