Wykres commitów

71 Commity (febd1abbd290001f49f42034447435f338abca23)

Autor SHA1 Wiadomość Data
Adam Laža f21b677083
OSM OpenMapTiles style ()
* OpenMapTiles style

* Rename style snippets from layer_name.json to defaultstyle.json

* Add README to icons dir.

* Add README to icons dir.

* Rename snippets to style.json

* Fix spritezero - move icons to svg subdir

* Move icons from style/icons/svg directly to style/icons

Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
2022-10-06 16:09:21 +02:00
Tomas Pohanka 029033a593
Filter a bigger areas for landcover z7 - z13 () 2022-09-01 15:02:15 +02:00
Tomas Pohanka dae00f2604
Add missing requires NE tables to layers. ()
Adding missing requires Natural Earth tables for layers.
2022-08-02 14:22:58 +02:00
Jeroen Hoek da7b1b1645
Add natural=shrubbery to landcover ()
This adds `natural=shrubbery` to landcover (next to the similar `natural=scrub`).

Fixes 
2022-06-01 18:20:35 +02:00
Falke Design ebeafc65fa
Add requires.tables to <layer>.yaml ()
**NOTE** this can only be merged after the next tools version is released.

Added required Postgres tables to the `<layer>.yaml` definition.
Close: 

PR of tools: https://github.com/openmaptiles/openmaptiles-tools/pull/370
2021-11-05 10:04:54 +01:00
Brian Sperlongano f744f9c009
Upgrade to omt-tools v6, Imposm 0.11.1 and migrate SQL ()
This PR migrates the SQL in the transportation and transportation_name layer to use the new imposm3 mappings which now map a separate primary key ID.
2021-10-14 11:38:28 -04:00
Matt Riggott 4d6945b935
Simplify landcover layer generalisation SQL ()
I was reading through the SQL used to generalise the landcover layer ([`layers/landcover/generalized.sql`](596f44aa26/layers/landcover/generalized.sql)) and I noticed that when creating the generalised landcover tables for zooms 9-13, the SQL was doing something like this:

```sql
SELECT * FROM simplify_vw_z13 WHERE ST_NPoints(geometry) < 50
UNION ALL
SELECT * FROM simplify_vw_z13 WHERE ST_NPoints(geometry) >= 50 AND ST_NPoints(geometry) < 300
```

As in, the simplification/clustering of landcover generalisations was being done in two steps: once for polygons with fewer than fifty points, and once for polygons with 50+ points.

As far as I can see, there's no benefit to doing this — it's probably just an artefact of the dev work done for the [original pull request](https://github.com/openmaptiles/openmaptiles/pull/1035). Of course, I might be completely missing the reason for clustering polygons into two groups (<50 points and 50+ points). But assuming I haven't, I've created this pull request to simplify the SQL used in the generalisation, merging the two steps into something like this:

```sql
SELECT * FROM simplify_vw_z13 WHERE ST_NPoints(geometry) < 300
```

The effect is to slightly reduce the number of landcover features (polygons with <50 points can now be clustered/unioned with those 50+), and to reduce the time taken to generalise the landcover tables (in a small test using OSM's Iceland data I saw a 7% reduction in the time spent). It doesn't alter the features that are shown on the map.

Builds upon work from commits 4a1b0afa26 and da689f9e42.
2021-10-08 08:51:20 +02:00
Brian Sperlongano 91c6d3bbd1
Update ETL docs on landcover layer ()
While reviewing , it became apparent that the ETL documentation for the landcover layer was incomplete.  This PR updates the ETL documentation in `generalized.sql` so that each table has a node in the ETL diagram.
2021-10-07 15:54:13 -04:00
Brian Sperlongano e8a3bc18c5
Remove landuse=park from landcover ()
The tag `landuse=park` is a rare tagging mistake of `leisure=park`.  The OSM wiki [lists this tag as a mistake](https://wiki.openstreetmap.org/wiki/Tag:leisure%3Dpark).  It is included in the landcover layer with a note that there were (at the time) 600 parks tagged in this way.  There are currently a bit over 100 objects tagged with this mistake.  Based on the wiki documentation and obvious decline of this tag, we should remove it so that we're not introducing mapping mistakes into the database and not inadvertently providing mapper feedback that encourages the use of mistake tagging.
2021-09-20 10:23:26 +02:00
Adam Laža 461043960e
Bugfix landcover osm ()
* Release 3.12 prerequisite

* Add missing osm_id column to osm_landcover_polygon

Co-authored-by: Tomáš <tompohys@gmail.com>
2021-01-08 15:44:11 +01:00
Adam Laža 5772e61244
Bugfix in osm_landcover_gen_z10. ()
Add missing `NOT` in WHERE condition.
It causes there are duplicated features of subclass `wood` and `forest` and features of other subclasses are missing.
2020-12-14 12:47:45 +02:00
Adam Laža 77f5d76e37
Refactor schema, one _gen_z view per zoom. ()
* Refactor layer aeroway.

* Refactor layer boundary.

* Refactor layer landcover.

* Refactor layer landuse.

* Refactor layer park.

* Refactor layer transportation.

* Refactor layer water.

* Refactor layer waterway.

* Re-generate water* layers etl_diagrams.

* Regenerate etl_diagrams for waterway.

* Cast NULL to text.
2020-11-26 13:54:00 +01:00
Adam Laža da689f9e42
Fix landcover generalization. ()
This PR brings few modification which make it possible to generalize landcover on larger areas. Tested on Europe.

- Use ST_SnapToGrid to avoid ERROR:  GEOSUnaryUnion: TopologyException: found non-noded intersection between LINESTRING.
- Cluster only landcover subclasses: wood and forest.
- Use less generalized tables as a starting point for next step generalization.
2020-11-24 10:33:06 +02:00
Tomas Pohanka 4a1b0afa26
reworked landcover layer ()
Instead of using [Douglas-Peucker algorithm](https://postgis.net/docs/ST_SimplifyPreserveTopology.html) which is using for generalized tables [imposm](https://imposm.org/docs/imposm3/latest/mapping.html#generalized-tables) it is used [Visvalingam-Whyatt algorithm](https://postgis.net/docs/ST_SimplifyVW.html).

Solution:
remove imposm generalized tables (during `import-osm`) and create generalization in `import-sql` step for zooms 7-14. Zooms 0-6 are from Natural Earth data.

Upper zoom levels (7 and 8) are ok to merge with no big impact on creation speed. In Canada from z9 it took too long. Between zoom levels 10 - 13 there can be union polygons with less than 300 edge points (empirical number based on a test on Canada forest, can be discussed). Polygons with more than 300 edge points are just simplified. Zoom 14 is from the original dataset `osm_landcover_polygon`.

There is also removed `osm_id` which is never used and not passed into vector tiles.
2020-11-11 15:11:48 +02:00
Yuri Astrakhan 2b95d1cffa
Fix & optimize incorrect function declarations ()
* All functions that access database must be declared as `STABLE`, not `IMMUTABLE` -- because database can change at any moment, e.g. during an update
* there are a few functions that could be made `STRICT` -- passing `NULL` as a parameter will always result in a `NULL`, but for some reason that causes a significant decrease in perf.
* tagged one function as parallel safe

NOTE: somehow `ST_AsMVT()` method of tile generation is showing 70-90% slowdown with this patch. I am not sure of why this is happening. If the reason is the `IMMUTABLE` -> `STABLE` change, we may have to dig deeper into PG optimization
2020-06-17 12:15:26 -04:00
Yuri Astrakhan 6457419e0d
NOOP: Format all layer's SQL code ()
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.

Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax

Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups

P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
2020-06-08 12:19:55 -04:00
Jorge Sanz ace759590e
Parallel capability to layer functions ()
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.

`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).

### Testing procedure

The procedure to test this was:

* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze 
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.

<details><summary>Testing query</summary>

```sql
-- Using the function layer_landuse
explain analyze 
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>

You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities

![image](https://user-images.githubusercontent.com/188264/70647153-9cac9300-1c48-11ea-96ea-ac7a1e2f4a79.png)

### Next steps

Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.

I will post my finding here, even no change in the code should happen.


cc. @nyurik

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2020-01-31 19:36:02 -05:00
Yuri Astrakhan 9d6dbfc64f
Use one pass docs image generation ()
quicker and cleaner diagram image generation.
Remove etl-graph and mapping-graph targets - redundant

Also, the obsolete "fields" is still in Imposm's code and both names are accepted,
but "fields" is not documented anywhere (PR submitted), and could be removed at any moment.

Our docs were not supporting it until this PR, so renaming it at the same time.

Several images have been updated due to a more inclusive mapping scan
Requires https://github.com/openmaptiles/openmaptiles-tools/pull/147 (merged)
2020-01-22 21:55:22 -05:00
Yuri Astrakhan c9e7ad90c6
Remove unneeded "else null" in conditions ()
Minor code cleanup:
SQL already returns NULL in the "WHEN" condition
if it is not matched by any of the cases.

Co-authored-by: Eva Jelinkova <evka.jelinkova@gmail.com>
2020-01-22 17:24:28 -05:00
Yuri Astrakhan 1614a4656c Manage field mapping in SQL declaratively
Simplify some of the OSM->OMT field value mappings using declarative syntax.

This approach is not for all cases, but in many it removes
the need of storing the same field in both the .yaml and .sql files.

TODO: support more complex AND/OR cases
2019-12-19 11:25:29 -05:00
Eva J 89815c04cf mapping_diagram update 2019-12-09 14:48:52 +01:00
Frédéric Rodrigo f916e29be0 Add leisure=golf_course, map it to grass 2019-11-29 13:43:10 +01:00
Eva J 90c4854184 removing webmerc_area 2019-11-19 16:36:00 +01:00
Eva Jelinkova 6ea4cde691
Merge branch 'master' into refactor-landcover 2019-11-05 12:02:37 +01:00
Frédéric Rodrigo a2917a9323 Map fell, heath, scrub and tundra like grassland, as 'grass' 2019-11-02 21:14:51 +01:00
Yuri Astrakhan 73c7290893 Updated mapping diagrams to match changes in landcover & transportation
These were forgotten to check in.  We should add validation to the tests
to make sure all values are up to date.
2019-11-01 22:40:39 -04:00
Eva Jelinkova e7aa43d91b
Merge pull request from frodrigo/dune
Support natural=dune aside of beach and sand
2019-10-29 15:00:20 +01:00
Yuri Astrakhan eca13f9bed
Refreshed all diagrams, fixed automation & 2 broken graphs ()
Seems like etl and mapping diagrams have been neglected
for a long time. Now it regenerates the files and places
them in the source dir.

This PR also fixes two broken files:
* layers/aerodrome_label/mapping_diagram.png
* layers/housenumber/mapping_diagram.png

They were generated using the newest tools version with the fix
https://github.com/openmaptiles/openmaptiles-tools/pull/65
2019-10-26 21:28:43 -04:00
Frédéric Rodrigo c6a13981ba Support natural=dune aside of beach and sand 2019-08-31 14:30:42 +02:00
François de Metz f4b56d00ae
Add a subclass column from the mapping_value to clean the landcover layer. 2019-07-28 12:36:01 +02:00
zstadler bf062d4731
Add missing `landcover` subclass values
Subclass values were extracted from `mapping.yaml`
2019-05-01 08:45:57 +03:00
jirik dba6e7025f Improve doc, update diagrams 2018-04-27 13:41:32 +02:00
Han Chao 974e85ff78 add class and subclass values to documentation 2018-02-01 09:25:38 +01:00
hanchao 51d4558ea8 add sand and beach 2018-02-01 09:25:38 +01:00
jirik 5c67f2769e Add st_isvalid check to imposm3 generalized tables
Fix 
2018-01-31 17:24:05 +01:00
jirik 9d803af8e3 Update landcover's documentation 2017-11-08 13:58:01 +01:00
jirik e1ab498936 Add gardens to landcover 2017-11-08 13:47:11 +01:00
stirringhalo 15ff070abd Add glaciers at higher zooms ()
* Add glaciers

* Add glacier to subclass
2017-06-25 22:10:09 -04:00
stirringhalo 2c309d7632 Add bare_rock and scree to sql 2017-06-25 18:56:54 -04:00
stirringhalo 112cf2b96b Add bare_rock scree 2017-06-25 16:50:20 -04:00
stirringhalo f6448f2501 Add landcover z7 2017-06-16 21:55:32 -04:00
jirik a68ccc42ee Adjust zres values for buildings,landcover and landuse
Decrease size of Belgium Z0-12 mbtiles from 63 MB to 38 MB
2017-05-22 19:03:44 +02:00
stirringhalo 90690d2a39 Switch to ZRes () 2017-04-25 18:03:29 -04:00
Jiri Kozel 3e6495a134 Import both area and webmerc_area, use area for filtering () 2017-03-24 16:46:21 +01:00
jirik 00dc515142 Update diagrams for landcover 2017-03-15 17:35:37 +01:00
jirik 70170b2e54 Add missing woods, fix 2017-03-15 15:35:31 +01:00
stirringhalo 8273bfbb62 Switch to webmercarea 2017-01-17 09:45:58 -05:00
Lukas Martinelli f361d3d9fd Add better title to layer READMEs 2017-01-05 17:31:21 +01:00
Lukas Martinelli 8a9b1246d3 Link to openmaptiles.org for layer doc 2017-01-05 17:29:18 +01:00
stirringhalo bab5c3d4b6 Switch from geom to geometry 2016-12-20 22:32:34 -05:00