mirror
/
openmaptiles
kopia lustrzana https://github.com/openmaptiles/openmaptiles
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
1 134 Commity
8 Gałęzie
21 Tagi
85 MiB
Wykres commitów

27 Commity (335e5b03ceed74cf1c5d9c56cccc4972a2a82311)

Autor SHA1 Wiadomość Data
Adam Laža 77f5d76e37
Refactor schema, one _gen_z view per zoom. (#1045) 
* Refactor layer aeroway.

* Refactor layer boundary.

* Refactor layer landcover.

* Refactor layer landuse.

* Refactor layer park.

* Refactor layer transportation.

* Refactor layer water.

* Refactor layer waterway.

* Re-generate water* layers etl_diagrams.

* Regenerate etl_diagrams for waterway.

* Cast NULL to text.
 2020-11-26 13:54:00 +01:00
Frédéric Rodrigo 9bb17792a6
Remove alignment of AS in SQL and few others (#932) 
* Remove alignment of AS in SQL

* Remove alignment of CREATE TABLE in SQL
 2020-07-22 13:48:25 +02:00
Yuri Astrakhan 2b95d1cffa
Fix & optimize incorrect function declarations (#918) 
* All functions that access database must be declared as `STABLE`, not `IMMUTABLE` -- because database can change at any moment, e.g. during an update
* there are a few functions that could be made `STRICT` -- passing `NULL` as a parameter will always result in a `NULL`, but for some reason that causes a significant decrease in perf.
* tagged one function as parallel safe

NOTE: somehow `ST_AsMVT()` method of tile generation is showing 70-90% slowdown with this patch. I am not sure of why this is happening. If the reason is the `IMMUTABLE` -> `STABLE` change, we may have to dig deeper into PG optimization
 2020-06-17 12:15:26 -04:00
Yuri Astrakhan 6457419e0d
NOOP: Format all layer's SQL code (#917) 
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.

Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax

Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups

P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
 2020-06-08 12:19:55 -04:00
Jorge Sanz ace759590e
Parallel capability to layer functions (#728) 
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
 2020-01-31 19:36:02 -05:00
Yuri Astrakhan c9e7ad90c6
Remove unneeded "else null" in conditions (#732) 
Minor code cleanup:
SQL already returns NULL in the "WHEN" condition
if it is not matched by any of the cases.

Co-authored-by: Eva Jelinkova <evka.jelinkova@gmail.com>
 2020-01-22 17:24:28 -05:00
zstadler 7a3ac079ae Add intermittent field to water* layers (#430 PR #585) 2019-05-24 13:21:43 +02:00
Frédéric Rodrigo 89ee0de23b Waterway: Rename function to waterway_brunnel, Remove ford. 2018-01-16 11:45:49 +01:00
Bob Wallis c246b3c4df Add brunnel to waterway layer 2018-01-16 11:45:49 +01:00
Jiri Kozel 51bc8fad35 Multilinguality (#279) 
Improve multilinguality: names in 57 languages, name:latin, name:nonlatin, name_int. Fixes #211 #252 #80.

See #279 for more info.
 2017-06-12 17:53:47 +02:00
jirik 4c6d30066d Add German names (name_de), unify English names (name_en) 2017-03-17 12:56:54 +01:00
Lukas Martinelli 5dbcc027df Group by name_en for important waterway 2017-01-18 16:18:50 +01:00
Lukas Martinelli 8e5caed2d8 Add name_en to waterway #88 2017-01-18 15:58:14 +01:00
stirringhalo bab5c3d4b6 Switch from geom to geometry 2016-12-20 22:32:34 -05:00
ImreSamu 087b2f204e fix SQL warning 2016-12-04 12:51:52 +01:00
ImreSamu f5ac3def5e etldoc fixes 2016-12-04 02:52:52 +01:00
Lukas Martinelli a93c066fec Add missing etldocs slots to waterway layer 2016-11-30 16:25:12 +00:00
Lukas Martinelli f9def56e0b Leave names empty for NE data due to different names 2016-11-29 15:08:23 +00:00
Lukas Martinelli 6f68ab999d Ensure empty name is NULL in waterway 2016-11-29 13:33:33 +00:00
Lukas Martinelli 5a679e1a9b Merge branch 'master' into waterway 
Conflicts:
	layers/waterway/waterway.sql
 2016-11-28 19:26:42 +00:00
stirringhalo 4f795b6287 Quiet down 'has type unknown' 2016-11-15 23:37:41 -05:00
lukasmartinelli 98d77b0638 Show only linestrings with name on z11 2016-11-15 21:23:54 +01:00
lukasmartinelli b77cc7a43a More nuanced waterway 2016-11-15 19:37:19 +01:00
ImreSamu 2a0ef3a953 add etldoc to layer_waterway 2016-11-10 15:00:13 +01:00
lukasmartinelli 0a8847d6ae Add class attribute to waterway 2016-10-24 15:38:32 +02:00
lukasmartinelli 15a61983ae Generalize waterway in imposm 2016-10-24 12:02:50 +02:00
lukasmartinelli 446544f5c8 Add waterway layer 2016-10-24 09:40:14 +02:00