Wykres commitów

47 Commity (44ae83718ca0727922f03c2aef39c2f3050051a8)

Autor SHA1 Wiadomość Data
Brian Sperlongano 0cff3449b5
Implement non-blocking refresh ()
This PR implements non-blocking updates in the park layer.

The approach was to use the `REFRESH MATERIALIZED VIEW CONCURRENTLY` feature in postgres.  In order to achieve this, a unique index was added on the z4 dissolved park area table.  The `ST_Union` / `ST_Dump` sequence was changed to an explicit cluster DB scan (an equivalent operation) so that a unique osm ID could be generated from each dissolved polygon.

Below is a screen shot from Idaho, USA showing that the dissolved z4 still works as expected.

![image](https://user-images.githubusercontent.com/3254090/142341513-588045f0-7757-4acd-99e5-a50bbe6b0682.png)
2021-11-24 11:17:29 +01:00
Brian Sperlongano cdff735be1
Remove blank name tags ()
This PR performs the same fix on the park layer that  applied to the waterway layer.  It removes blank name, name_en, and name_de from the tiles.
2021-11-18 18:27:19 +01:00
Brian Sperlongano 9e74fb6299
Fix park layer ETL diagram ()
This PR fixes an ETL diagram documentation error in the park layer.
2021-10-25 08:12:17 -04:00
Brian Sperlongano f744f9c009
Upgrade to omt-tools v6, Imposm 0.11.1 and migrate SQL ()
This PR migrates the SQL in the transportation and transportation_name layer to use the new imposm3 mappings which now map a separate primary key ID.
2021-10-14 11:38:28 -04:00
Brian Sperlongano 718e79a704
Fix park polygon update SQL ()
This PR fixes a bug in the park polygon update code introduced in .  Because park polygons at zoom 4 are dissolved, the centroid is not present in the table at that layer.  Thus, a separate update trigger is needed to avoid errors.  This PR adds the separate trigger for the zoom 4 park layer.

This PR likely resolves an unidentified blocker for 
2021-09-29 13:40:16 +02:00
Brian Sperlongano 0e17d53f42
Implement park rendering at z4-5 ()
Unblocks 

Currently, OpenMapTiles renders protected areas at zoom 6 and higher.  However, the western United States has protected areas that are so vast that it makes sense to render these areas as low as zoom 4, as noted in 

This PR extends the generalization of park features from its current termination at zoom 6 to zoom 4.

Here is an example from the Americana style of rendering protected areas at zoom 6, which is currently the lowest supported zoom:
https://zelonewolf.github.io/openstreetmap-americana/#6/45.313/-111.721

We would like to provide rendering at zoom 4 (no labels) and zoom 5 (with labels) in the Americana style.  Below are samples generated from this PR for Idaho, USA:

Zoom 4:
![image](https://user-images.githubusercontent.com/3254090/126924222-e21865e9-b184-479a-abd5-46238bd106f8.png)

Zoom 5:
![image](https://user-images.githubusercontent.com/3254090/126924230-01853e8c-d3e0-41e8-b10f-06415da529d8.png)
2021-09-01 07:20:52 +02:00
Adam Laža 77f5d76e37
Refactor schema, one _gen_z view per zoom. ()
* Refactor layer aeroway.

* Refactor layer boundary.

* Refactor layer landcover.

* Refactor layer landuse.

* Refactor layer park.

* Refactor layer transportation.

* Refactor layer water.

* Refactor layer waterway.

* Re-generate water* layers etl_diagrams.

* Regenerate etl_diagrams for waterway.

* Cast NULL to text.
2020-11-26 13:54:00 +01:00
Adam Laža 0dc8c3256c
Rename generic layer.sql to actual_layer_name .sql ()
Following layers had generic `layer.sql` name.

- `aerodrome_label`
- `aeroway`
- `housenumber`
- `mountain_peak`
- `park`
- `place`
- `poi`
- `transportation`
- `transportation_name`
- `water_name`

This PR renames `layer.sql` to `aerodrome_label.sql`, `aeroway.sql`...
2020-11-09 11:43:35 +02:00
Frédéric Rodrigo 9bb17792a6
Remove alignment of AS in SQL and few others ()
* Remove alignment of AS in SQL

* Remove alignment of CREATE TABLE in SQL
2020-07-22 13:48:25 +02:00
Yuri Astrakhan 2b95d1cffa
Fix & optimize incorrect function declarations ()
* All functions that access database must be declared as `STABLE`, not `IMMUTABLE` -- because database can change at any moment, e.g. during an update
* there are a few functions that could be made `STRICT` -- passing `NULL` as a parameter will always result in a `NULL`, but for some reason that causes a significant decrease in perf.
* tagged one function as parallel safe

NOTE: somehow `ST_AsMVT()` method of tile generation is showing 70-90% slowdown with this patch. I am not sure of why this is happening. If the reason is the `IMMUTABLE` -> `STABLE` change, we may have to dig deeper into PG optimization
2020-06-17 12:15:26 -04:00
zstadler 5e65ac8c52
Add area-base filtering for park names in lower zoom levels ()
Resolve https://github.com/openmaptiles/openmaptiles/issues/776

Thanks @zstadler for this PR.
2020-06-09 10:53:18 +02:00
Yuri Astrakhan 6457419e0d
NOOP: Format all layer's SQL code ()
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.

Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax

Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups

P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
2020-06-08 12:19:55 -04:00
zstadler 2584489ad7
Continue conflict resolution 2020-04-04 13:16:59 +03:00
zstadler 88e9127bfd
Merge branch 'master' into protected_area 2020-04-04 12:50:03 +03:00
Ludovic Delauné 924ccd6015
remove useless sql selection () 2020-03-13 15:14:30 -04:00
Yuri Astrakhan 2edbbfa1a0 Updated etl diagrams from latest tools
* gets rid of duplicate connecting lines in etl graphs
2020-02-14 16:22:48 -05:00
Jorge Sanz ace759590e
Parallel capability to layer functions ()
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2020-01-31 19:36:02 -05:00
zstadler 5657ede450 Add `boundary=protected_area` parks
Update the **class** of the **park* layer in a backward-compatible way.

The class for `boundary=protected_area` parks is the lower-case of the
[`protection_title`](http://wiki.openstreetmap.org/wiki/key:protection_title)
value with blanks replaced by `_`.
`national_park` is the class of `protection_title=National Park` and `boundary=national_park`.
`nature_reserve` is the class of `protection_title=Nature Reserve` and `leisure=nature_reserve`.
The class for other [`protection_title`](http://wiki.openstreetmap.org/wiki/key:protection_title)
values is similarly assigned.

Resolve https://github.com/openmaptiles/openmaptiles/issues/760
2020-01-27 11:12:57 +02:00
Yuri Astrakhan 9d6dbfc64f
Use one pass docs image generation ()
quicker and cleaner diagram image generation.
Remove etl-graph and mapping-graph targets - redundant

Also, the obsolete "fields" is still in Imposm's code and both names are accepted,
but "fields" is not documented anywhere (PR submitted), and could be removed at any moment.

Our docs were not supporting it until this PR, so renaming it at the same time.

Several images have been updated due to a more inclusive mapping scan
Requires https://github.com/openmaptiles/openmaptiles-tools/pull/147 (merged)
2020-01-22 21:55:22 -05:00
Yuri Astrakhan e6e92036a0
Use _resolve_wikidata in layer mapping.yaml ()
Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.

This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.

The next tools version will remove it when generating imposm mapping file.
2020-01-21 09:43:21 -05:00
Eva J 90c4854184 removing webmerc_area 2019-11-19 16:36:00 +01:00
Yuri Astrakhan eca13f9bed
Refreshed all diagrams, fixed automation & 2 broken graphs ()
Seems like etl and mapping diagrams have been neglected
for a long time. Now it regenerates the files and places
them in the source dir.

This PR also fixes two broken files:
* layers/aerodrome_label/mapping_diagram.png
* layers/housenumber/mapping_diagram.png

They were generated using the newest tools version with the fix
https://github.com/openmaptiles/openmaptiles-tools/pull/65
2019-10-26 21:28:43 -04:00
Adrien Matissart 993d0ef589 Fix typo in update_osm_park_polygon_row function 2018-12-19 17:58:42 +01:00
jirik 927d3945e2 Add geometry_point indexes to park layer 2018-11-30 15:32:26 +01:00
jirik fe057eede1 Move geometry && bbox check in park layer 2018-11-30 15:32:26 +01:00
jirik 7cd29e2622 Add extra geometry_point && bbox check to park layer 2018-11-30 15:32:26 +01:00
jirik cd920755bf Remove park_label layer, add labels to park layer 2018-11-07 18:22:26 +01:00
jirik ccf8ee6dfa Add park labels layer 2018-11-05 16:40:32 +01:00
jirik 5c67f2769e Add st_isvalid check to imposm3 generalized tables
Fix 
2018-01-31 17:24:05 +01:00
jirik c2c5fc8f32 Generate park_polygon_gen8 from _gen7 2018-01-31 17:24:05 +01:00
jirik 3cee3c7f80 Update diagrams 2017-12-04 12:06:28 +01:00
Jiri Kozel f848b0449b Adjust park tolerance ()
* Adjust st_length filter based on zres in transportation layer

Probably related to 

* Adjust park tolerance

Related to , 
2017-05-25 12:05:12 +02:00
jirik 4037256d43 Eliminate features with area<4px for aeroways, parks and water 2017-05-22 19:03:44 +02:00
stirringhalo 90690d2a39 Switch to ZRes () 2017-04-25 18:03:29 -04:00
Jiri Kozel 3e6495a134 Import both area and webmerc_area, use area for filtering () 2017-03-24 16:46:21 +01:00
stirringhalo 8273bfbb62 Switch to webmercarea 2017-01-17 09:45:58 -05:00
Lukas Martinelli f361d3d9fd Add better title to layer READMEs 2017-01-05 17:31:21 +01:00
Lukas Martinelli 8a9b1246d3 Link to openmaptiles.org for layer doc 2017-01-05 17:29:18 +01:00
Lukas Martinelli 5ec4f0766b Add zoom level range slots to landcover diagram 2016-12-01 08:29:34 +00:00
Lukas Martinelli c1b1913c7c Fix wrong etl docs 2016-11-30 18:38:28 +00:00
Lukas Martinelli 8523c91a6b Remove no longer needed park class functions 2016-11-30 18:30:52 +00:00
Lukas Martinelli bcc38a0911 Show large parks already on z6 2016-11-30 17:11:47 +00:00
Lukas Martinelli 0c3b3b8b63 Show large parks on z7 2016-11-30 14:16:09 +00:00
Lukas Martinelli 915cd2bde3 Add possible values for landcover, landuse, park doc 2016-11-30 11:21:13 +00:00
Lukas Martinelli 4a902ca273 Extend docs for park, landuse, landcover 2016-11-30 09:32:25 +00:00
Lukas Martinelli 5ebb33575e Move public parks indicating grass to landcover 2016-11-29 19:38:02 +00:00
Lukas Martinelli 09dddebac4 Add park layer 2016-11-28 19:16:27 +00:00