2019-03-25 18:55:39 +00:00
# Introduction
2023-03-06 18:14:01 +00:00
Thank you for considering contributing to OpenMapTiles. It's people like you that make OpenMapTiles such a great project. Talk to us at the OSM Slack ** #openmaptiles ** channel ([join](https://slack.openstreetmap.us/)).
2019-03-25 18:55:39 +00:00
Following these guidelines helps to communicate that you respect the time of the developers managing and developing this open source project. In return, they should reciprocate that respect in addressing your issue, assessing changes, and helping you finalize your pull requests.
OpenMapTiles is an open source project and we love to receive contributions from our community — you! There are many ways to contribute, from writing tutorials or blog posts, improving the documentation, submitting bug reports and feature requests or writing code which can be incorporated into OpenMapTiles itself.
# Ground Rules
* Create issues for any major changes and enhancements that you wish to make. Discuss things transparently and get community feedback.
* Keep feature versions as small as possible, preferably one new feature per version.
* Be welcoming to newcomers and encourage diverse new contributors from all backgrounds. See the [Python Community Code of Conduct ](https://www.python.org/psf/codeofconduct/ ).
# Getting started
1. Create your own fork of the code
1. Do the changes in your fork
1. Create a pull request
# Code review process
2019-05-20 12:14:25 +00:00
We all make mistakes and bad coding decisions. So apart from the obvious fixes, all changes must be reviewed by another 2 members of the project. This also helps with the [bus factor ](https://en.wikipedia.org/wiki/Bus_factor ) -- there should always be other people in the team who know why a change was made.
2019-03-25 18:55:39 +00:00
2019-05-20 12:14:25 +00:00
For any non-trivial changes, all pull requests must be approved by at least three members of the OpenMapTiles team. Afterwards you can merge the PR if you have rights, or another person must do it for you.
2019-03-25 18:55:39 +00:00
2019-05-20 12:14:25 +00:00
Your pull request must:
2019-03-25 18:55:39 +00:00
2019-05-20 12:14:25 +00:00
* Address a single issue or add a single item of functionality.
* Contain a clean history of small, incremental, logically separate commits,
with no merge commits.
* Use clear commit messages.
* Be possible to merge automatically.
2020-01-23 02:55:22 +00:00
When you modify import data rules in `mapping.yaml` or `*.sql` , please update:
2019-05-20 12:14:25 +00:00
1. field description in `[layer].yaml`
2. comments starting with `#etldoc`
2020-01-23 02:55:22 +00:00
3. regenerate documentation graphs with `make generate-devdoc`
2019-05-20 12:14:25 +00:00
4. update layer description on https://openmaptiles.org/schema/ (https://github.com/openmaptiles/www.openmaptiles.org/tree/master/layers)
5. check if OMT styles are affected by the PR and if there is a need for style updates
When you are making PR that adds new spatial features to OpenMapTiles schema, please make also PR for at least one of our GL styles to show it on the map. Visual check is crucial.
Framework for SQL-based unit tests for import and updates (#1249)
This PR adds the ability to create SQL tests that ensure that OSM data is properly imported and updated in the OpenMapTiles data schema. The tests work by injecting test OSM data and updates into the database and checking to ensure that the data is properly loaded into the database using standard SQL statements. With this framework in place, developers can now write small tests to inject known data into the database and ensure that imports and updates are working correctly.
In addition to the framework, basic tests are provided for four layers. These initial tests are in no way comprehensive, but they provide a structure and framework for developers to add their own tests or expand the existing ones to cover more cases.
Usage:
`make clean && make sql-test`
## How it works
The SQL tests consist of the following parts:
1. **Test import data**, located in `tests/import`. This test data is in the [OSM XML](https://wiki.openstreetmap.org/wiki/OSM_XML) format and contains the data that should be initially injected into the database. The files are numbered in order to ensure that each test data file contains OSM id numbers that are distinct from the other files. For example, the file starting with `100` will use node ids from 100000-199999, way ids from 1000-1999, and relation ids from 100-199.
1. **Test update data**, located in `tests/update`. This test data is in the [osmChange XML](https://wiki.openstreetmap.org/wiki/OsmChange) format, and contains the data that will be used to update the test import data (in order to verify that the update process is working correctly. These files are also numbered using the same scheme as the test import data.
1. **Import SQL test script**, located at `tests/test-post-import.sql`. This script is executed after the test import data has been injected, and runs SQL-based checks to ensure that the import data was properly imported. If there are failures in the tests, an entry will be added to the table `omt_test_failures`, with one record per error that occurs during the import process. A test failure will also fail the build. To inspect the test failure messages, run `make psql` and issue the comment `SELECT * FROM omt_test_failures`.
1. **Update SQL test script**, located at `tests/test-post-update.sql`. This script performs the same function as the import test script, except that it occurs after the test update data has been applied to the database. Note that script will only run if the import script passes all tests.
2021-10-28 11:38:46 +00:00
2021-11-01 09:43:37 +00:00
# SQL unit testing
Framework for SQL-based unit tests for import and updates (#1249)
This PR adds the ability to create SQL tests that ensure that OSM data is properly imported and updated in the OpenMapTiles data schema. The tests work by injecting test OSM data and updates into the database and checking to ensure that the data is properly loaded into the database using standard SQL statements. With this framework in place, developers can now write small tests to inject known data into the database and ensure that imports and updates are working correctly.
In addition to the framework, basic tests are provided for four layers. These initial tests are in no way comprehensive, but they provide a structure and framework for developers to add their own tests or expand the existing ones to cover more cases.
Usage:
`make clean && make sql-test`
## How it works
The SQL tests consist of the following parts:
1. **Test import data**, located in `tests/import`. This test data is in the [OSM XML](https://wiki.openstreetmap.org/wiki/OSM_XML) format and contains the data that should be initially injected into the database. The files are numbered in order to ensure that each test data file contains OSM id numbers that are distinct from the other files. For example, the file starting with `100` will use node ids from 100000-199999, way ids from 1000-1999, and relation ids from 100-199.
1. **Test update data**, located in `tests/update`. This test data is in the [osmChange XML](https://wiki.openstreetmap.org/wiki/OsmChange) format, and contains the data that will be used to update the test import data (in order to verify that the update process is working correctly. These files are also numbered using the same scheme as the test import data.
1. **Import SQL test script**, located at `tests/test-post-import.sql`. This script is executed after the test import data has been injected, and runs SQL-based checks to ensure that the import data was properly imported. If there are failures in the tests, an entry will be added to the table `omt_test_failures`, with one record per error that occurs during the import process. A test failure will also fail the build. To inspect the test failure messages, run `make psql` and issue the comment `SELECT * FROM omt_test_failures`.
1. **Update SQL test script**, located at `tests/test-post-update.sql`. This script performs the same function as the import test script, except that it occurs after the test update data has been applied to the database. Note that script will only run if the import script passes all tests.
2021-10-28 11:38:46 +00:00
It is recommended that you create a [unit test ](TESTING.md ) when modifying the behavior of the SQL layer. This will ensure that your changes are working as expected when importing or updating OSM data into an OpenMapTiles database.
2021-11-01 09:43:37 +00:00
# Verifying that updates still work
2021-11-16 06:58:16 +00:00
When testing a PR, you should also verify that the update process completes without an error. Please modify, if necessary, and run the script below.
2021-11-01 09:43:37 +00:00
2021-11-16 06:58:16 +00:00
**Note:**
The verification requires the script to append temporary changes to the `.env` file. Please restore the original version from git using `git checkout .env` or remove these changes before submitting a PR.
2021-11-01 09:43:37 +00:00
```
2021-11-16 06:58:16 +00:00
(
set -e
2021-11-01 09:43:37 +00:00
2021-11-16 06:58:16 +00:00
cat >> .env < < EOM
2021-11-01 09:43:37 +00:00
2021-11-16 06:58:16 +00:00
# temporary changes for verifying that updates still work
# Ensure DIFF_MODE is active
DIFF_MODE=true
# Ensure all zoom levels are tested
MAX_ZOOM=14
EOM
# Set the test area to the appropriate geofabrik extract
export area=north-america/us/indiana
# Build 1-month-old tiles
2021-11-01 09:43:37 +00:00
rm -fr data build cache
2021-11-16 06:58:16 +00:00
make destroy-db
2021-11-01 09:43:37 +00:00
make download-geofabrik area=$area
2021-11-16 06:58:16 +00:00
docker-compose run --rm --user=$(id -u):$(id -g) openmaptiles-tools sh -c "wget -nv -O data/$area.osm.pbf http://download.geofabrik.de/$area-$(date --date="$(date +%Y-%m-15) -1 month" +'%y%m01').osm.pbf"
2021-11-01 09:43:37 +00:00
./quickstart.sh $area
2021-11-16 06:58:16 +00:00
cat < < EOM
# Update with the changes since a month+ ago
EOM
2021-11-01 09:43:37 +00:00
docker-compose run --rm --user=$(id -u):$(id -g) openmaptiles-tools sh -c "osmupdate --base-url=$(sed -n 's/ *\"replication_url\": //p' data/$area.repl.json) data/$area.osm.pbf data/changes.osc.gz"
make import-diff
make generate-tiles-pg
2021-11-16 06:58:16 +00:00
) < /dev/null
2021-11-01 09:43:37 +00:00
```