This PR replaces the `route_N` attribute scheme with individual attributes for name and colour on routes. Thus you will have:
* `route_N_network` to hold the route `network` value
* `route_N_ref` to hold the route `ref` value
* `route_N_name` to hold the route `name` value
* `route_N_colour` to hold the route `colour` or `ref:colour` value
Adding source_ids column if not exist in the table
- osm_transportation_merge_linestring_gen_z11
- osm_transportation_merge_linestring_gen_z8
- osm_transportation_name_linestring
- osm_important_waterway_linestring
This PR addresses two main issues introduced by the new ID-Based Merged-LineString Updates
1. Partial Indexes can only be accessed when the query matches more or less exactly and the query-planner will fail to use indexes when targeted via the join-condition and not the where-condition
- `osm_transportation_merge_linestring_gen_z9`
- `osm_transportation_name_network`
- `osm_shipway_linestring`
- `osm_aerialway_linestring`
- `osm_waterway_linestring`
2. When intersecting updated Source-LineStrings with the existing Merged-LineStrings we join the Source-IDs of each existing Merged-LineString. This bloats the table unnecessarily and slows down bigger updates considerably.
- This is addressed by aggregating the Source-IDs of each existing Merged-LineString into an array and concatinating these arrays when grouping them. Afterwards we add the IDs of updated SourceLineStrings and deduplicate the result before adding it to the Source-IDs-Table.
LEAKPROOF requires the executing user to have superuser privileges when creating the functions which would be the only statement required to be executed as a superuser.
This PR now removes bridge/tunnel/ford attributes from the `transportation` layer from sections of bridge/tunnel etc that are too small to be meaningfully visible at a particular zoom level. This allows those segments to be generalized into geometries along with the surrounding sections of road, thereby dropping lots of useless little pieces of geometry in lower-zoom tiles.
This PR changes the three-column unique index to an on-insert trigger. This should fix the issues we're having with CI failures and still achieve the behavior of coalescing duplicate routes.
I moved the concurrency_index calculation into an intermediate materialized view to separate the de-duplication capability from both DENSE_RANK() and from imposm updates.
This PR defines the set of routes in Canada which are equivalent in national importance to the US Interstate Highway System, and are therefore appropriate to render at zoom 4. This creates a sane, connected highway network at this zoom level across the USA and Canada.
This adds two additional network types for Canadian highways, ca-provincial for provincial-level roads, and ca-provincial-arterial for "highest importance" roads that are not part of the Trans-Canada highway but should be regarded as equivalent for low-zoom rendering purposes.
Additionally, this extracts out the country-specific network checks to a separate function in order to define "equivalent top-level networks" by country, providing a place to add additional national definitions as they're added by contributors.
This PR improves the behavior introduced in #1440, which inadvertently included all motorways at z4, which was unintended. Instead, this PR provides an appropriate progression from z4 to z5 with the highest national-importance highways being rendered at z4 (including where they degrade to trunk, as in Canada), and all motorways being added to the mix at z5.
This adds support to show the Trans-Canada Highway at zoom 4. Despite being the most important highway network in Canada, portions are `highway=trunk` due to the remoteness of the countryside. However, it's still important to show a connected highway network at the lowest zoom without showing all trunk roads at this zoom.
This change also adds support for US Interstate Highways to be drawn at z4 when they're tagged as trunks. There's only a few examples of this, but it removes those (tiny) gaps from the generalized road geometries. As we identify additional countries which should have their most important road network rendered at the lowest road zoom even if they're trunk, we can add them to the list of networks that get this treatment.
Guided busways (or bus guideways in OSM terminology) are special roadways along which bus operators can travel smoothly at high speeds without steering. These are tagged highway=bus_guideway on OpenStreetMap.
Currently, highway=busway is surfaced in OpenMapTiles as the busway class in the transportation layer, but highway=bus_guideway is not present. These two types of roadways serve generally similar purposes, so it would make sense to have guided busways at the same zoom levels.
Replace the `REFRESH MATERIALIZED` on `osm_transportation_merge_linestring_gen_z*` by differential update.
The way of doing this is the same as other differential updates. But in this case there two `GROUP BY` level. So all the the process have to be done twice.
PR move `get_basic_names(tags, geometry)` function from `transportation_name_tags` function and adding it after `transportation_name_tags` resolve street abbrevation.
PR runs through `test-sql` test without error. Adding to `osm_transportation_name_network` (table and for update) was done in consequence of `test-sql` errors ([update test 500](a747d98550/tests/test-post-update.sql (L53))). Originally `get_basic_names` was just in `osm_transportation_name_linestring` table.
FIX#1324
Fixes#1148
This PR adds expressway tagging to the `transportation` layer, by setting `expressway=1` for non-motorway roads tagged `expressway=yes`, and omitting the tag otherwise. Additionally, I've added a few unit tests to verify that the expressway tagging is being imported and updated into the intermediate tables.
Here is an example of expressway tagging on US-1 in Rhode Island, USA:
I discovered this bug while investigating issues with the updates process related to #1190#1292, and #814.
The `transportation_name` layer produces slightly different `tags` hstore values in the `osm_transportation_name_linestring` table during the initial import versus when running an update. As currently written, the import code produces null-value keys in the `tags` column, while the update code suppresses them. This PR removes that difference and makes the import code use same method that is currently used in the update code.
With a test case I've written, the import code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name:de"=>NULL, "name:en"=>NULL, "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`
...while the update code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`
Note the missing NULL values.
This bug causes a small amount of space wastage after an update is run, because the update matching code detects the `tags` value as different, resulting in a duplicate copy of the tags value if that row is updated. This causes duplicate objects and breaks GROUP BY clauses that expect to group same-tagged features together. I've tested this by inspection of a generated mbtiles, database spot checks, and the unit test code included in this PR.
Closes#271
This PR adds track and path rendering at lower zooms than currently provided, and also achieves near-parity with openstreetmap-carto on track and path rendering. A previously-abandoned attempt, with significant discussion, was #1169.
This PR fixes a bug in the `WHERE` clauses in the transportation layer generalization tables for zoom 10. The intended behavior of the layer is to suppress `highway=tertiary` and `highway=tertiary_link` at zoom 10 and lower. However, due to this bug, these objects were not suppressed as intended, because an `OR` was used in the SQL where an `AND` was needed instead.
This bug was inadvertently introduced in #1172😞
This PR removes unneeded `CASE` clauses which converts `toll` tags to booleans. However, the `toll` tag is already mapped as a boolean by imposm, therefore this processing can be simplified.
Fixes#366
This PR sets `toll=1` in the `transportation` layer when a road is tagged as a toll road in OSM (with the tag `toll=yes`). If a road is tagged with any other value of `tag=*`, the value is suppressed in the tile, since non-toll roads can be presumed as the default and therefore this PR should have only negligible impact in the tiles.
Support for toll road tagging is of interest in the American mapping community, because toll roads have historically been styled differently on American-style maps, for example:
Screen shot of a toll road being generated in the `transportation` layer tiles:
Fixes#1066
This PR adds a new field `access` in the transportation layer, which will be set to `no` if the `access` tag is either `no` or `private`. While `private` is the more popular value by a 17:1 ratio, I went with `no` because it's smaller in the tiles. In addition, the text `no` opens up the future possibility of other text-based access values such as `destination`, `customers`, or `permit`.
The screenshot below shows an example of access tagging for a road on a military base:
This PR updates the `transportation` layer creation scripts to simplify the SQL and remove unneeded sub-selects, checks and conditionals, fix indexes, and improve inline documentation.
Currently, there are two sequences of materialized view creations. There is one from zoom 11 through 9, and a second one for zoom 8 through 4. This PR removes that break in the sequence of transportation table materialized view creations, in favor of one in which high-zoom views are created first, and then each lower zoom is created from the zoom above.
Instead, the current generalized zoom 8 transportation table is built directly from `osm_transportation_linestring` rather than being built from the zoom 9 transportation table. This means that when building the zoom 8 table, it must scan the entire transportation network rather than just selecting from the pre-filtered zoom 9 table.
This PR removes an unneeded sub-select in the build of the zoom 8 table, which appears to be a leftover from an old version of the SQL that did some sort of merge. Once this PR is implemented all zooms from 11 through 4 will be linked via a progressive series of materialized views.
Lastly, this adds in missing materialized view refreshes for zooms 9-11, which appear to have been entirely missing, and as far as I can tell aren't getting updated in the current version of this code.
In addition, the following optimizations were added as part of this commit:
1. Updates the `osm_highway_linestring_highway_partial_idx` partial index to match the `SELECT..WHERE` clause actually present in `transportation/update_route_member.sql`, which is where it appears to be actually used, and update inline documentation to reflect this.
2. Collapses unnecessary sub-select block in `osm_transportation_merge_linestring_gen_z11`, and removes unnecessary ST_IsValid() call, which already provided in `mapping.yaml`, and update inline documentation to reflect these assumptions.
3. Updates `WHERE` blocks to remove unnecesary checks and further document assumptions. The `highway=construction` check is unnecessary in all cases, because it is sufficient to check the `construction` key alone. If `construction=*` is set, then `highway=construction` is implied.
4. Two indexes were added to `layers/transportation/update_route_member.sql` to improve route population performance.
In testing locally, I'm seeing performance improvements around 10% in the generation of the `transportation` layer, based on modifying `openmaptiles.yaml` to generate only the transportation layer and then repeatedly running `time make import-sql`, however, this timing might be impacted by docker, so I would ask for confirmation of acceptable performance.
In addition, this PR shortens the length of the transportation update SQL file by 30 lines, which makes it easier for contributors to work with.
PR #1168 removed several `WHERE` clauses in the `transportation_merge_*` table series. With those removed, it appears that `osm_transportation_merge_linestring_gen_z8` and `osm_transportation_merge_linestring` are nearly identical, with the former simply adding an `ST_Simplify()` operation.
A grep of the codebase indicates that the _only_ use for `osm_transportation_merge_linestring` is to hold the intermediate result of an `ST_Dump(geometry))` before it is fed into `ST_Simplify()`. Therefore, it appears that we're holding an entire zoom 8 copy of the transportation layer (all motorway/trunk/primary roads) in a materialized view for absolutely no reason at all.
This PR removes the `osm_transportation_merge_linestring` intermediate table and changes the definition of `osm_transportation_merge_linestring_gen_z8` to perform the `ST_Dump()` and `ST_Simplify()` transformations directly from the `osm_highway_linestring` table in a single operation.
This PR removes several redundant/unnecessary WHERE clauses from the transportation layer. Specifically:
The table `osm_transportation_merge_linestring` is a view of `osm_highway_linestring` which exposes only motorway/trunk/primary roads:
9e4be3e3b0/layers/transportation/update_transportation_merge.sql (L122-L123)
However, the create statement for the table `osm_transportation_merge_linestring_gen_z8`, which is a view of `osm_transportation_merge_linestring`, also contains a `WHERE` clause which selects down to motorway/trunk/primary roads. This `WHERE` is unnecessary:
9e4be3e3b0/layers/transportation/update_transportation_merge.sql (L144-L145)
This unneeded `WHERE` clause is similarly present in the create statement for `osm_transportation_merge_linestring_gen_z7`, which is a view of `osm_transportation_merge_linestring_gen_z8`:
9e4be3e3b0/layers/transportation/update_transportation_merge.sql (L163-L164)
Likewise, there is a similar redundant `WHERE` clause in the `osm_transportation_merge_linestring_gen_z5` and `osm_transportation_merge_linestring_gen_z6` tables, both of which select down to `motorway` and `trunk`. This `WHERE` clause is only needed on the z6 table, and is redundant on the z5 table.
I am not sure what the performance penalty is for these redundant `WHERE` clauses, but there does not appear to be any reason to keep them, and they may incur a performance cost.
To avoid discontinuous transportation lines between zooms 9 and 11.
- Originally limit geometry by length for z9 - z11 (`ST_Length(geometry) > ZRes(11)`)
- highway z9 to z11 was generalized during import-osm
- now just create a filtered and generalized z11 table
- then merge segments in the same way as from (full-featured) osm_highway_linestring and used this merged z11 for mat.view z10 and z9
Close#1107
On creation of materialized views `osm_transportation_merge_linestring_gen_zX`, an index is created each time the new view and not used else where than the next generalization step.
It does not worth it to to a scan on the table to create index that will be used only once, while the next generalisation step can also be done in one full scan.
So, removing these indices.
The index `osm_highway_linestring_highway_idx` is not used. There is also a used partial index `osm_highway_linestring_highway_partial_idx`.
I cross check the usage of the index from postgres query stats and all SQL query code where the table `osm_highway_linestring` is refered.
This index is relatively big.
I would like to reformat all of our SQL to have a concise coding style.
This makes it far easier to understand the code for a casual contributor,
and lets us spot errors more easily.
Most importantly, it makes it much easier to grep (search) the code because it is more likely to be in the same syntax
Some key changes:
* SQL keywords are always UPPERCASE, e.g. `SELECT WHEN AS END ...`
* types, variables, aliases, and field names (identifiers) are always lower case
* `LANGUAGE 'plpgsql'` is now `LANGUAGE plpgsql` (no quotes)
* a few minor spacing/semicolon cleanups
P.S. Per @TomPohys request, `TABLE` is spelled using upper case despite being a type for consistency with PG Docs. Same for `LANGUAGE SQL` vs `LANGUAGE plpgsql`.
Tag all SQL materialized views with a machine-readable comment
to indicate that this materialized view can be created without
data:
/* DELAY_MATERIALIZED_VIEW_CREATION */
In the next version of tools this comment can be optionally
replaced with the "WITH NO DATA" parameter, thus allowing
a much faster execution of the SQL script. All materialized
viewes will be populated with data in parallel afterwards
using the `refresh-views` tools script.
Brian Sperlongano
