25 Commity (83f8a23bd3de79deeae34e81d7c27a84865b1e22)

Autor	SHA1	Wiadomość	Data
Jorge Sanz	ace759590e	Parallel capability to layer functions (#728) ... This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced. `PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html). ### Testing procedure The procedure to test this was: * Imported `spain.pbf` in a clean environment * Dumped the OpenMapTiles database from the Postgres Docker image * Created a clean Postgres 12 database using the default Docker image * Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.) * Restored the dump * Lowered the postgres planner parameters for triggering parallel plans: ```sql set parallel_setup_cost = 5; set parallel_tuple_cost = 0.005; ``` * Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions). * For each layer, run a testing query to confirm parallel workers were created, something like this: ```sql explain analyze select * from layer_aerodrome_label(tilebbox(8,128,95),10,null) union all select * from layer_aerodrome_label(tilebbox(8,128,97),10,null); ``` * After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function. <details><summary>Testing query</summary> ```sql -- Using the function layer_landuse explain analyze select geometry from layer_water(tilebbox(14,8020,6178),14) union all select geometry from layer_waterway(tilebbox(14,8020,6178),14) union all select geometry from layer_landcover(tilebbox(14,8020,6178),14) union all select geometry from layer_landuse(tilebbox(14,8020,6178),14) union all select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14) union all select geometry from layer_park(tilebbox(14,8020,6178),14) union all select geometry from layer_boundary(tilebbox(14,8020,6178),14) union all select geometry from layer_aeroway(tilebbox(14,8020,6178),14) union all select geometry from layer_transportation(tilebbox(14,8020,6178),14) union all select geometry from layer_building(tilebbox(14,8020,6178),14) union all select geometry from layer_water_name(tilebbox(14,8020,6178),14) union all select geometry from layer_transportation_name(tilebbox(14,8020,6178),14) union all select geometry from layer_place(tilebbox(14,8020,6178),14) union all select geometry from layer_housenumber(tilebbox(14,8020,6178),14) union all select geometry from layer_poi(tilebbox(14,8020,6178),14) union all select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14); ``` </details> You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities  ### Next steps Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request. I will post my finding here, even no change in the code should happen. cc. @nyurik Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>	2020-01-31 19:36:02 -05:00
Yuri Astrakhan	1bc3f138b4	Create boundary materialized views (#740) ... * Move simplified border tables to OMT as MAT VIEWs Consolidate derived table creation in OMT repository. Move all non-original simplified geometry tables from import-osmborder image to this repo, allowing further optimization. Later we can remove derived table creation fro mthe import-osmborder image.	2020-01-21 09:44:29 -05:00
Eva J	4c43c6ea3b	add data types for boundary_z0	2019-12-18 19:31:28 +01:00
Eva J	6381451964	just newlines	2019-12-12 09:48:35 +01:00
Eva J	fee22d4eb9	adding disp. relations into z1-4	2019-12-11 12:05:42 +01:00
Eva J	0181513b1a	2 attributes, generalization in mapping, edit unions	2019-12-10 19:48:24 +01:00
Eva J	fd940684eb	relation member values, diagrams	2019-12-10 08:49:54 +01:00
Eva J	a05228381a	add relations with claimed_by	2019-12-07 12:18:06 +01:00
Eva J	062e2c89b0	adding disputed for NE sources	2019-11-07 18:07:15 +01:00
Eva J	8f543c5e62	removing using scalerank	2019-05-06 18:04:29 +02:00
Eva J	9146fc74b0	changes in z4 and z5	2019-05-03 14:47:30 +02:00
Eric Theise	7e625de3e1	Correct etldoc mapping.	2018-11-05 10:01:06 +01:00
jirik	26cc6dbbea	Remove borders of leased areas at Z4	2018-04-27 12:44:04 +02:00
jirik	77c1c038d5	Remove borders of leased areas ... Fix #416. Affects Baykonur and Guantanamo.	2018-04-16 09:18:13 +02:00
jirik	db60ae7f40	Fix missing state and province boundaries at Z4 and Z5	2017-12-04 11:00:53 +01:00
antoine-de	2ddbc8b1c1	update the layers from natural earth latest data ... Natural earth has released new data and now some sql query are invalid should fix #352 I don't really know how natural earth data versioning is maintained though 😕	2017-11-17 13:55:02 +01:00
Lukas Martinelli	d9f791e11e	Mix maritime boundaries with NE and switch to OSM earlier	2017-01-16 20:50:56 +01:00
Lukas Martinelli	8874140207	Use boundaries from osmborder	2017-01-16 20:11:35 +01:00
stirringhalo	bab5c3d4b6	Switch from geom to geometry	2016-12-20 22:32:34 -05:00
ImreSamu	f5ac3def5e	etldoc fixes	2016-12-04 02:52:52 +01:00
Lukas Martinelli	1f27b72a1a	Show more province boundaries on z6	2016-11-30 16:56:34 +00:00
ImreSamu	598c931e17	boundary ETL graph comments - only sql tables and views	2016-11-09 22:06:24 +01:00
lukasmartinelli	ff52cfda08	Remove unnecessary fields from boundary	2016-10-28 20:39:38 +02:00
lukasmartinelli	28fedcc021	Add imposm3 mapping for boundary	2016-10-10 19:29:50 +02:00
lukasmartinelli	480d720585	Move to new modular layer structure	2016-10-09 21:27:09 +02:00