openmaptiles

OpenMapTiles is an extensible and open tile schema based on the OpenStreetMap.

maps openstreetmap openstreetmap-data tiles vector-tiles

Go to file

Jorge Sanz ace759590e Parallel capability to layer functions (#728 ) This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced. `PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html). ### Testing procedure The procedure to test this was: * Imported `spain.pbf` in a clean environment * Dumped the OpenMapTiles database from the Postgres Docker image * Created a clean Postgres 12 database using the default Docker image * Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.) * Restored the dump * Lowered the postgres planner parameters for triggering parallel plans: ```sql set parallel_setup_cost = 5; set parallel_tuple_cost = 0.005; ``` * Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions). * For each layer, run a testing query to confirm parallel workers were created, something like this: ```sql explain analyze select * from layer_aerodrome_label(tilebbox(8,128,95),10,null) union all select * from layer_aerodrome_label(tilebbox(8,128,97),10,null); ``` * After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function. <details><summary>Testing query</summary> ```sql -- Using the function layer_landuse explain analyze select geometry from layer_water(tilebbox(14,8020,6178),14) union all select geometry from layer_waterway(tilebbox(14,8020,6178),14) union all select geometry from layer_landcover(tilebbox(14,8020,6178),14) union all select geometry from layer_landuse(tilebbox(14,8020,6178),14) union all select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14) union all select geometry from layer_park(tilebbox(14,8020,6178),14) union all select geometry from layer_boundary(tilebbox(14,8020,6178),14) union all select geometry from layer_aeroway(tilebbox(14,8020,6178),14) union all select geometry from layer_transportation(tilebbox(14,8020,6178),14) union all select geometry from layer_building(tilebbox(14,8020,6178),14) union all select geometry from layer_water_name(tilebbox(14,8020,6178),14) union all select geometry from layer_transportation_name(tilebbox(14,8020,6178),14) union all select geometry from layer_place(tilebbox(14,8020,6178),14) union all select geometry from layer_housenumber(tilebbox(14,8020,6178),14) union all select geometry from layer_poi(tilebbox(14,8020,6178),14) union all select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14); ``` </details> You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities ![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png) ### Next steps Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request. I will post my finding here, even no change in the code should happen. cc. @nyurik Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>		2020-01-31 19:36:02 -05:00
.github	Fix typo in pull_request_template (#682 )	2019-10-21 22:31:24 -04:00
layers	Parallel capability to layer functions (#728 )	2020-01-31 19:36:02 -05:00
qa	Removed TOOLS_VERSION, rewritten commands to use only docker-compose with TOOLS_VERSION inside .env.	2019-12-16 15:37:19 +01:00
.env	Fixed docker-compose commands, missing default TOOLS_VERSION.	2019-12-13 14:41:39 +01:00
.gitignore	Migrate to new Wikidata importer (#735 )	2020-01-22 16:16:38 -05:00
.travis.yml	TravisCI should raise error on fail	2019-12-16 14:07:53 -05:00
BENCHMARKING.md	fix formatting	2019-02-06 18:17:40 -05:00
CONTRIBUTING.md	Use one pass docs image generation (#751 )	2020-01-22 21:55:22 -05:00
LICENSE.md	Create LICENSE.md	2017-01-07 15:10:07 +01:00
Makefile	Declared field mapping 2 (#734 )	2020-01-31 00:22:21 -05:00
QUICKSTART.md	Minor trailing whitespace cleanup (#596 )	2019-03-29 17:15:42 -04:00
README.md	Migrate to new Wikidata importer (#735 )	2020-01-22 16:16:38 -05:00
UPDATE.md	Use later images (#289 )	2017-06-24 18:11:34 -04:00
docker-compose.yml	Remove unused import-sql image (#738 )	2020-01-24 14:40:03 -05:00
openmaptiles.yaml	OMT version update	2019-12-13 09:39:40 +01:00
quickstart.sh	Remove unused import-sql image (#738 )	2020-01-24 14:40:03 -05:00

README.md

OpenMapTiles

OpenMapTiles is an extensible and open tile schema based on the OpenStreetMap. This project is used to generate vector tiles for online zoomable maps. OpenMapTiles is about creating a beautiful basemaps with general layers containing topographic information. More information openmaptiles.org and openmaptiles.com.

We encourage you to collaborate, reuse and adapt existing layers, or add your own layers. You may use our approach for your own vector tile project. Feel free to fork the repo and experiment. The repository is built on top of the openmaptiles/openmaptiles-tools to simplify vector tile creation.

Please keep in mind that OpenMapTiles schema should display general topographic content. If creating a new layer or expanding an existing layer with a specific theme, please create a fork and invite other community members to cooperate on your topic. OpenMapTiles schema is used in many projects all over the world and the size of the final vector tiles needs to be considered in any update.

🔗 Schema https://openmaptiles.org/schema
🔗 Docs https://openmaptiles.org/docs
🔗 Production package: https://openmaptiles.com/production-package/
🔗 Hosting https://www.maptiler.com/cloud/
🔗 Create own layer https://github.com/openmaptiles/openmaptiles-skiing
🔗 Discuss at the #openmaptiles channel at OSM Slack

Styles

You can start from several GL styles supporting the OpenMapTiles vector schema.

🔗 Learn how to create Mapbox GL styles with Maputnik and OpenMapTiles.

We also ported over our favorite old raster styles (TM2).

🔗 Learn how to create TM2 styles with Mapbox Studio Classic and OpenMapTiles.

Schema

OpenMapTiles consists out of a collection of documented and self contained layers you can modify and adapt. Together the layers make up the OpenMapTiles tileset.

🔗 Study the vector tile schema

Develop

To work on OpenMapTiles you need Docker.

Install Docker. Minimum version is 1.12.3+.
Install Docker Compose. Minimum version is 1.7.1+.

Build

Build the tileset.

git clone https://github.com/openmaptiles/openmaptiles.git
cd openmaptiles
# Build the imposm mapping, the tm2source project and collect all SQL scripts
make

You can execute the following manual steps (for better understanding) or use the provided quickstart.sh script.

./quickstart.sh

Prepare the Database

Now start up the database container.

docker-compose up -d postgres

Import external data from OpenStreetMapData, Natural Earth and OpenStreetMap Lake Labels.

docker-compose run import-water
docker-compose run import-natural-earth
docker-compose run import-lakelines
docker-compose run import-osmborder

Download OpenStreetMap data extracts and store the PBF file in the ./data directory.

cd data
wget http://download.geofabrik.de/europe/albania-latest.osm.pbf

Import OpenStreetMap data with the mapping rules from build/mapping.yaml (which has been created by make).

docker-compose run import-osm

Import latest Wikidata. If an OSM feature has Key:wikidata, OpenMapTiles check corresponding item in Wikidata and use its labels for languages listed in openmaptiles.yaml. So the generated vector tiles includes multi-languages in name field.

This step uses Wikidata Query Service to download just the Wikidata IDs that already exist in the database.

make import-wikidata

Work on Layers

Each time you modify layer SQL code run make and make import-sql.

make clean
make
make import-sql

Now you are ready to generate the vector tiles. Using environment variables you can limit the bounding box and zoom levels of what you want to generate (docker-compose.yml).

docker-compose run generate-vectortiles

License

All code in this repository is under the BSD license and the cartography decisions encoded in the schema and SQL are licensed under CC-BY.

Products or services using maps derived from OpenMapTiles schema need to visibly credit "OpenMapTiles.org" or reference "OpenMapTiles" with a link to https://openmaptiles.org/. Exceptions to attribution requirement can be granted on request.

For a browsable electronic map based on OpenMapTiles and OpenStreetMap data, the credit should appear in the corner of the map. For example:

For printed and static maps a similar attribution should be made in a textual description near the image, in the same fashion as if you cite a photograph.