* OpenMapTiles style
* Rename style snippets from layer_name.json to defaultstyle.json
* Add README to icons dir.
* Add README to icons dir.
* Rename snippets to style.json
* Fix spritezero - move icons to svg subdir
* Move icons from style/icons/svg directly to style/icons
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
Guided busways (or bus guideways in OSM terminology) are special roadways along which bus operators can travel smoothly at high speeds without steering. These are tagged highway=bus_guideway on OpenStreetMap.
Currently, highway=busway is surfaced in OpenMapTiles as the busway class in the transportation layer, but highway=bus_guideway is not present. These two types of roadways serve generally similar purposes, so it would make sense to have guided busways at the same zoom levels.
Before this, only `aerialway=cable_car` and `aerialway=gondola` were included in the tiles. That misses some widely-used tags, notably `aerialway=chair_lift`, `aerialway=drag_lift`, and `aerialway=platter`. The usage of all the tags added in this commit, by fraction of all aerialway way keys (according to TagInfo), is:
| Tag | Usage |
|------------------------|-------:|
| `aerialway=chair_lift` | 25.51% |
| `aerialway=drag_lift` | 12.93% |
| `aerialway=platter` | 11.35% |
| `aerialway=t-bar` | 7.93% |
| `aerialway=j-bar` | 0.63% |
| `aerialway=mixed_lift` | 0.24% |
Some of these tags were also added in PR #620 back in May 2019, but were removed again before it was merged.
Fixes#1342.
Updates on `transporation_name` layer take much more time than before.
In 3.13 there had been introduced highway concurrency into `transportation` and `transportation_name` so I expected the update process will take more time but not that much. Because of this it's impossible to use updates on larger areas because the process takes too long.
The issue is caused by `IS NOT DISTINCT FROM` operator over `tags` (hstore) columns. I replaced it with `=` operator in combination with `coalesce()` function which returns the same results but in shorter time.
PR move `get_basic_names(tags, geometry)` function from `transportation_name_tags` function and adding it after `transportation_name_tags` resolve street abbrevation.
PR runs through `test-sql` test without error. Adding to `osm_transportation_name_network` (table and for update) was done in consequence of `test-sql` errors ([update test 500](a747d98550/tests/test-post-update.sql (L53))). Originally `get_basic_names` was just in `osm_transportation_name_linestring` table.
FIX#1324
I discovered this bug while investigating issues with the updates process related to #1190#1292, and #814.
The `transportation_name` layer produces slightly different `tags` hstore values in the `osm_transportation_name_linestring` table during the initial import versus when running an update. As currently written, the import code produces null-value keys in the `tags` column, while the update code suppresses them. This PR removes that difference and makes the import code use same method that is currently used in the update code.
With a test case I've written, the import code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name:de"=>NULL, "name:en"=>NULL, "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`
...while the update code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`
Note the missing NULL values.
This bug causes a small amount of space wastage after an update is run, because the update matching code detects the `tags` value as different, resulting in a duplicate copy of the tags value if that row is updated. This causes duplicate objects and breaks GROUP BY clauses that expect to group same-tagged features together. I've tested this by inspection of a generated mbtiles, database spot checks, and the unit test code included in this PR.
Closes#271
This PR adds track and path rendering at lower zooms than currently provided, and also achieves near-parity with openstreetmap-carto on track and path rendering. A previously-abandoned attempt, with significant discussion, was #1169.
This PR is a bugfix for the `transportation_name` layer. Currently, the updates process does not work, as noted by @zstadler in #1230. The issue is that the computed `concurrency_index` column in `osm_route_member` is never updated after the initial updates. Therefore, whenever route member ways are updated, they will lose their route concurrency index, and hence route concurrencies will be lost in the `transportation_name` tiles. This PR adds the missing concurrency index update code to the transportation/network update triggers.
This PR as written also incorporates the fixes in #1230 and #1233.
The following SQL is a unit test to demonstrate that these triggers work correctly:
```
select count(*) from osm_route_member where concurrency_index > 5;
select count(*) from osm_transportation_name_network where route_6 is not null;
update osm_route_member set concurrency_index=NULL where concurrency_index > 5;
select count(*) from osm_route_member where concurrency_index > 5;
select count(*) from osm_transportation_name_network where route_6 is not null;
```
If working correctly, both pairs of `count(*)` values should return the same pair of numbers.
While troubleshooting #1230, I discovered that there were cases where a way was a member of the same route relation more than once, such as: https://www.openstreetmap.org/way/17439235
<img src="https://user-images.githubusercontent.com/3254090/133726616-acd01332-343e-4930-ad3d-1fb3c3190a4c.png" width=280/>
This segment would end up showing duplicate highway shields in a rendered style, which is not desirable. While I personally think this style of tagging is wrong, it's a relatively easy fix to remove those duplicates. This PR replaces `ROW_NUMBER()` with `DENSE_RANK()` in the concurrency index generation code and adds a `DISTINCT` constraint on the concurrency join to handle multiple member rows with the same concurrency index.
Since I was in this file, I also fixed the copy/paste error in generating the route names.
Screenshot from database showing correct mapping:
Fixes#1191
This PR suppresses `highway=service` at zoom 12, where it is not a useful level of detail. This makes OpenMapTiles consistent with openstreetmap-carto, which does not begin showing `highway=service` until raster zoom 14 / vector zoom 13.
Additionally, this PR suppresses `highway=service` + `service=parking_aisle` / `service=driveway` from zooms 12-13, as this detail is excessive below zoom 14. As a point of comparison, openstreetmap-carto does not begin showing `service=parking_aisle` / `service=driveway` until raster zoom 16 (vector zoom 15).
Changes in #1143 removed the need for `name` and `ref` parameters in the network-to-network_type conversion function. This PR removes this dead code.
Verified by running ./quickstart rhode-island which completed successfully.
This PR fixed bugs introduced in #1147 and #1119 which broke the update triggers in `transportation_name_update`. I noticed this issue while I was working a different PR and tried to update a table in the `transportation` layer.
**Test**
The following code currently fails because of cascading update failures, but will complete successfully after this PR is merged.
```
UPDATE osm_highway_linestring hl
SET network = rm.network_type
FROM osm_route_member rm
WHERE hl.osm_id=rm.member;
```
Fixes#1128
This PR adds 6 new columns to the `transportation_name` column, named `route_1` through `route_6`. These columns contain route information for a section of roadway. The value is stored in the form **network=ref**. For example, Interstate 95 in the United States would be `US:I=95`. Thus, each `route_N` value contains enough information to render a highway shield. Since a section of road can be a part of more than one route, the `route_2`, `route_3`, etc, will contain the 2nd, 3rd, etc., concurrent routes.
The technical approach was to extend the change in #1135, which added ordered concurrency indexes to the `osm_route_member` table by joining up to the the first six entries to the `osm_transportation_name_network` table. In addition, that PR provided a ranking system for concurrent highways, ordering first by `network_type` (for example, `us-interstate`, `us-state`, etc), then alphabetically by network name, and then by ref in ascending order. This ordering of concurrent route memberships is now exposed in this PR in the sequential `route_N` values, meaning that rendered concurrent highway shields will be reasonably sorted.
The renderings below were generated using this branch of OpenMapTiles, as well as a separate branch of openstreetmap-americana:
https://github.com/ZeLonewolf/openstreetmap-americana/tree/openmaptilers-new-features-test
The rendering approach is to use the [formatted expressions](https://maplibre.org/maplibre-gl-js-docs/style-spec/expressions/) feature in mapLibre to insert images into a string of text. Blank shields are added to the sprite sheet for all possible route networks. Next, a [styleimagemissing](https://maplibre.org/maplibre-gl-js-docs/api/map/) callback is registered. As each shield ID is requested, the callback retrieves the sprite shield blank associated with the route's network, draw the `ref` text on the shield, and insert the complete shield back into the map.
Of note, this approach currently results in shields which are rotated about the road rather than being viewport aligned. This issue is currently documented as maplibre/maplibre-gl-js#188. A separate repository (https://github.com/ZeLonewolf/maplibre-shield-rotation-sample) has been created as a test case to fix this rotation issue.
Adding route concurrency information to OpenMapTiles would be a major step forward in achieving comprehensive highway shield renderings in a vector map!
**Renderings**:
Fixes#1131
This change does the following:
1. Excludes roads from `transportation_name` that don't have a `name` or a `ref`
2. Updates the road name merging logic to exclude changes in `brunnel` status. This will ensure that minor bridges don't disrupt the continuity of named roads as the map zooms out.
3. The `brunnel` tag will now only be set when a bridge or tunnel is distinctly named. Distinctly named is defined as "has a different name from the road on either side".
This example shows an unnamed interstate highway rendered as a continuous feature at low zoom. This road has many small bridges along its length:
This example shows a named bridge rendered with `brunnel` tag set:
Closes#1134
Adds a `concurrency_index` column to `osm_route_member` as described in #1134. For example, listing concurrencies on a [segment of the Washington Bridge](https://www.openstreetmap.org/way/43080535) in Providence, Rhode Island, USA:
```
openmaptiles=# select osm_id, network, ref, network_type, name, concurrency_index from osm_route_member where member=43080535 order by concurrency_index asc;
osm_id | network | ref | network_type | name | concurrency_index
----------+---------+-----+---------------+---------------+-------------------
-1694950 | US:I | 195 | us-interstate | I 195 (RI/MA) | 1
-2308410 | US:US | 6 | us-highway | US 6 (RI) | 2
-1347856 | US:US | 1A | us-highway | US 1A (RI) | 3
-2309143 | US:US | 44 | us-highway | US 44 (RI) | 4
(4 rows)
```
This PR adds a layer for `highway=motorway_junction` features.
This implementation of highway exits in the transportation_name layer add to the existing layer table structure, and renames the internal column name from "construction" (which was already overloaded with non-construction usages) to "subclass", which will be less confusing to future developers. The string 'junction' is used as the universal sub-class for highway exits.
A new documentation PR has been opened at openmaptiles/www.openmaptiles.org#69 to reflect these changes in the documentation.
To avoid discontinuous transportation lines between zooms 9 and 11.
- Originally limit geometry by length for z9 - z11 (`ST_Length(geometry) > ZRes(11)`)
- highway z9 to z11 was generalized during import-osm
- now just create a filtered and generalized z11 table
- then merge segments in the same way as from (full-featured) osm_highway_linestring and used this merged z11 for mat.view z10 and z9
Close#1107
https://github.com/openmaptiles/openmaptiles/issues/999
- Point 1 solved only partially - there is `brunnel` value added into zooms 9, 10, 11. For zooms 8 and smaller it would need change in osm_transportation_merge_ tables - I am going to try it but maybe its SQL costs will be to high.
- Point 2 solved.
Replacing materialized view by a tables with update from trigger on change only.
Differential update of `osm_transportation_name_linestring`, `osm_transportation_name_linestring` and `osm_transportation_name_linestring_genX`).
At the end of the transaction the dependent rows are updated.
The goal is to update more quickly the content of derivated tables by just updating the changing content. It replaces the update of materialized view because their need a full recompute (with lock issue).
It is the last part of the replacement of materialized view for the transportation layer.
It addresses #814 and a part of #809.
When OSM roads in the `transportation_name` layer are stitched together, their grouping does not consider all `name:*` tags.
As a result, roads with different `name:*` tags may be stitched together.
The `waterway` layer performs the grouping properly, for the same purpose:
1685eaccbd/layers/waterway/update_important_waterway.sql (L34)
Co-authored-by: Frédéric Rodrigo <fred.rodrigo@gmail.com>
Reduce the size of the materialized view osm_transportation_name_network by moving the weare clause.
osm_transportation_name_network is only used for osm_transportation_name_linestring.
Co-authored-by: zstadler <zeev.stadler@gmail.com>
Replacing materialized view by a tables with update from trigger on change only.
The osm_id of object changing on table osm_route_member and osm_highway_linestring are logged.
At the end of the transaction the dependent row of osm_transportation_name_network are updated.
The goal is to update more quickly the content of derivated tables by just updating the changing content. It replaces the update of materialized view because their need a full recompute (with lock issue).
Note, it is only a part of the materialized view of the transportation layer, I am also working on the other parts, other parts are in #892 (omaterialized view on sm_transportation_name_linestring and osm_transportation_name_linestring_genX).
It addresses #814 and a part of #809.
Clean and do refactoring on `update_route_member.sql` as first step to replace materialized view osm_transportation_name_network by tables with diff update. See #892.
* All functions that access database must be declared as `STABLE`, not `IMMUTABLE` -- because database can change at any moment, e.g. during an update
* there are a few functions that could be made `STRICT` -- passing `NULL` as a parameter will always result in a `NULL`, but for some reason that causes a significant decrease in perf.
* tagged one function as parallel safe
NOTE: somehow `ST_AsMVT()` method of tile generation is showing 70-90% slowdown with this patch. I am not sure of why this is happening. If the reason is the `IMMUTABLE` -> `STABLE` change, we may have to dig deeper into PG optimization
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.
Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax
Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups
P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
Mark waterway and transoprtation_name as having a dependency on another layer.
This is currently an unused parameter, but tools will use it later for faster
sql code generation.
Closes#796
This PR allows queries to be parallelized on recent versions of Postgres. The `PARALLEL SAFE` modifier has been added to the layer functions and a PLPGSQL function to convert strings into number has been replaced.
`PARALLEL SAFE` is a modifier for `CREATE FUNCTION` available since Postgres 9.6, so this change does not break current OpenMapTiles supported database version. More details about this topic [here](https://www.postgresql.org/docs/current/parallel-safety.html) and at the reference documentation for [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html).
### Testing procedure
The procedure to test this was:
* Imported `spain.pbf` in a clean environment
* Dumped the OpenMapTiles database from the Postgres Docker image
* Created a clean Postgres 12 database using the default Docker image
* Installed `postgis` 3 from the default Debian package and `osml10n` 2.5.8 from the repository (`make`, etc.)
* Restored the dump
* Lowered the postgres planner parameters for triggering parallel plans:
```sql
set parallel_setup_cost = 5;
set parallel_tuple_cost = 0.005;
```
* Manually added the `PARALLEL SAFE` modifier to each function involved in layer queries (not on updates or inserting functions).
* For each layer, run a testing query to confirm parallel workers were created, something like this:
```sql
explain analyze
select * from layer_aerodrome_label(tilebbox(8,128,95),10,null)
union all
select * from layer_aerodrome_label(tilebbox(8,128,97),10,null);
```
* After all the layers were processed and confirmed to start parallel executions, a more complete example was run. This example just retrieves the geometries for all the layers from the same tile but without using any MVT related function.
<details><summary>Testing query</summary>
```sql
-- Using the function layer_landuse
explain analyze
select geometry from layer_water(tilebbox(14,8020,6178),14)
union all
select geometry from layer_waterway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landcover(tilebbox(14,8020,6178),14)
union all
select geometry from layer_landuse(tilebbox(14,8020,6178),14)
union all
select geometry from layer_mountain_peak(tilebbox(14,8020,6178),14)
union all
select geometry from layer_park(tilebbox(14,8020,6178),14)
union all
select geometry from layer_boundary(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aeroway(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation(tilebbox(14,8020,6178),14)
union all
select geometry from layer_building(tilebbox(14,8020,6178),14)
union all
select geometry from layer_water_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_transportation_name(tilebbox(14,8020,6178),14)
union all
select geometry from layer_place(tilebbox(14,8020,6178),14)
union all
select geometry from layer_housenumber(tilebbox(14,8020,6178),14)
union all
select geometry from layer_poi(tilebbox(14,8020,6178),14)
union all
select geometry from layer_aerodrome_label(tilebbox(14,8020,6178),14);
```
</details>
You can inspect the execution plan and results on [this page](https://explain.dalibo.com/plan/3z). Also [attaching](https://github.com/openmaptiles/openmaptiles/files/3951822/explain-tile-simple.tar.gz) the query and JSON output for future reference. The website gives a ton of details, but you may want to search for nodes mentioning `workers` or `parallel` like in this area referring to `osm_border` or `osm_aeroway_linestring` entities
### Next steps
Since the execution plan is not showing a parallel append at the top level, meaning it's not running each layer individually, I want to continue experimenting with parameters and queries to see if it's possible to even parallelize more the request.
I will post my finding here, even no change in the code should happen.
cc. @nyurik
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
quicker and cleaner diagram image generation.
Remove etl-graph and mapping-graph targets - redundant
Also, the obsolete "fields" is still in Imposm's code and both names are accepted,
but "fields" is not documented anywhere (PR submitted), and could be removed at any moment.
Our docs were not supporting it until this PR, so renaming it at the same time.
Several images have been updated due to a more inclusive mapping scan
Requires https://github.com/openmaptiles/openmaptiles-tools/pull/147 (merged)
Minor code cleanup:
SQL already returns NULL in the "WHEN" condition
if it is not matched by any of the cases.
Co-authored-by: Eva Jelinkova <evka.jelinkova@gmail.com>
Adam Laža
