This PR removes an unnecessary check in water.yml.
generated SQL before:
```
SELECT CASE
WHEN waterway='riverbank' THEN 'river'
WHEN "waterway" = 'dock' THEN 'dock'
WHEN "water" = 'river'
OR "waterway" = 'riverbank'
THEN 'river'
WHEN "waterway" = '' THEN 'lake'
ELSE 'lake'
END;
```
generated SQL after:
```
SELECT CASE
WHEN waterway='riverbank' THEN 'river'
WHEN "waterway" = 'dock' THEN 'dock'
WHEN "water" = 'river'
OR "waterway" = 'riverbank'
THEN 'river'
ELSE 'lake'
END;
```
The `waterway=''` test is purely redundant because it returns the same value as the fall-through ELSE case: `lake`.
Make targets `start-tileserver`, `start-maputnik`, and `update-osm` initialize on-going processes rather than perform a one-time task. It is more appropriate to execute them using `docker-compose up -d`. With the current `docker-compose run` execution, the user needs to run `make` as a background process. This PR also allows the use of `docker-compose logs` rather than the need to handle `stdout` and `stderr` redirection and avoids the dependency of `stdin` handling for `docker-compose run`
Using `docker-compose` rather than `docker`
Use appropriate` start-*` and `stop-*` make target names
Resolves#1352
Before this, only `aerialway=cable_car` and `aerialway=gondola` were included in the tiles. That misses some widely-used tags, notably `aerialway=chair_lift`, `aerialway=drag_lift`, and `aerialway=platter`. The usage of all the tags added in this commit, by fraction of all aerialway way keys (according to TagInfo), is:
| Tag | Usage |
|------------------------|-------:|
| `aerialway=chair_lift` | 25.51% |
| `aerialway=drag_lift` | 12.93% |
| `aerialway=platter` | 11.35% |
| `aerialway=t-bar` | 7.93% |
| `aerialway=j-bar` | 0.63% |
| `aerialway=mixed_lift` | 0.24% |
Some of these tags were also added in PR #620 back in May 2019, but were removed again before it was merged.
Fixes#1342.
Updates on `transporation_name` layer take much more time than before.
In 3.13 there had been introduced highway concurrency into `transportation` and `transportation_name` so I expected the update process will take more time but not that much. Because of this it's impossible to use updates on larger areas because the process takes too long.
The issue is caused by `IS NOT DISTINCT FROM` operator over `tags` (hstore) columns. I replaced it with `=` operator in combination with `coalesce()` function which returns the same results but in shorter time.
When docker compose returns a version with v prefix,
the version check would incorrectly treat it as version component
of 000, leading to always failing the docker-compose version check
even if the version is recent enough.
This happens at least with docker-compose 2.2.1 where
docker-compose version --short
returns v2.2.1 with the leading v.
This PR fixes a bug that causes that `track` lines disappear at z13. This bug was introduced in https://github.com/openmaptiles/openmaptiles/pull/1190, which adds rendering of paths and tracks at z12 and z13.
Before this PR:
z12: lines with `route_rank = 1` are added (no matter what `highway`).
z13: lines with `route_rank BETWEEN 1 AND 2` and `highway = 'path'` are added.
-> tracks with `route_rank=1` are added at z12 but not at z13.
After this PR
z12: lines with `route_rank = 1` and `highway IN ('path', 'track')` are added.
z13: lines with `route_rank BETWEEN 1 AND 2` and `highway IN ('path', 'track')` are added .
-> only tracks and paths are added at z12 and z13 (which was IMHO the goal of https://github.com/openmaptiles/openmaptiles/pull/1190)
* Add only the most important paths and tracks (route_rank=1) to z12 and more important ones (route_rank between 1-2 or sac_scale or has name) to z13.
Remove `FILTER_MAPNIK_OUTPUT` variables from docker-compose which stop raising warnings during start-up dockers.
This functionality was removed in https://github.com/openmaptiles/openmaptiles-tools/pull/349.
The current version of the mapnik does not need `FILTER_MAPNIK_OUTPUT`.
PR move `get_basic_names(tags, geometry)` function from `transportation_name_tags` function and adding it after `transportation_name_tags` resolve street abbrevation.
PR runs through `test-sql` test without error. Adding to `osm_transportation_name_network` (table and for update) was done in consequence of `test-sql` errors ([update test 500](a747d98550/tests/test-post-update.sql (L53))). Originally `get_basic_names` was just in `osm_transportation_name_linestring` table.
FIX#1324
Remove `STRICT` statement from `transportation_name_tags` function.
Concurrently with remove `STRICT` from `get_basic_names` and `get_latin_name` in OMT-T
FIX#1319
* Add info about using MID_ZOOM to optimize tile generation
* Remove params for generation that are now automatically detected
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
Fixes#1148
This PR adds expressway tagging to the `transportation` layer, by setting `expressway=1` for non-motorway roads tagged `expressway=yes`, and omitting the tag otherwise. Additionally, I've added a few unit tests to verify that the expressway tagging is being imported and updated into the intermediate tables.
Here is an example of expressway tagging on US-1 in Rhode Island, USA:
I discovered this bug while investigating issues with the updates process related to #1190#1292, and #814.
The `transportation_name` layer produces slightly different `tags` hstore values in the `osm_transportation_name_linestring` table during the initial import versus when running an update. As currently written, the import code produces null-value keys in the `tags` column, while the update code suppresses them. This PR removes that difference and makes the import code use same method that is currently used in the update code.
With a test case I've written, the import code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name:de"=>NULL, "name:en"=>NULL, "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`
...while the update code produces a tags hstore that looks like this:
`"name"=>"OpenMapTiles Secondary 2", "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"`
Note the missing NULL values.
This bug causes a small amount of space wastage after an update is run, because the update matching code detects the `tags` value as different, resulting in a duplicate copy of the tags value if that row is updated. This causes duplicate objects and breaks GROUP BY clauses that expect to group same-tagged features together. I've tested this by inspection of a generated mbtiles, database spot checks, and the unit test code included in this PR.
This PR implements non-blocking updates in the park layer.
The approach was to use the `REFRESH MATERIALIZED VIEW CONCURRENTLY` feature in postgres. In order to achieve this, a unique index was added on the z4 dissolved park area table. The `ST_Union` / `ST_Dump` sequence was changed to an explicit cluster DB scan (an equivalent operation) so that a unique osm ID could be generated from each dissolved polygon.
Below is a screen shot from Idaho, USA showing that the dissolved z4 still works as expected.
Closes#271
This PR adds track and path rendering at lower zooms than currently provided, and also achieves near-parity with openstreetmap-carto on track and path rendering. A previously-abandoned attempt, with significant discussion, was #1169.
This PR adds unit tests for the transportation layer, and fixes several bugs in the unit testing code. It verifies the correct zoom association to the various highway classes, and verifies that updates to highway attributes are populated.
Additionally, the documentation for unit tests in the CONTRIB guide is updated to note the specific make commands for developers to run.
This PR is solving https://github.com/openmaptiles/openmaptiles/issues/406, partially also https://github.com/openmaptiles/openmaptiles/issues/320.
It is changing source of data for zoomlevel 6 - 8 in `waterway` layer to OSM (relation `waterway = river`), this way it corresponds with `water` layer.
Testing on Europe, z0 - z9 tiles increase size from 20.7 MB to 21.4 MB. We can discuss the level of generalization, but I think this size change is not critical.
This PR fixes a bug in the `waterway` layer in which empty `name_en` and `name_de` tags were being added to the tiles.
`name_en` and `name_de` are legacy tags, intended to be removed in v4.0. Thus, any legacy consumers that have not yet transitioned to `name:en` and `name:de` etc are currently NOT rendering labels for these features when `name:en` and `name:de` are not tagged on the feature. Therefore there are two possible approaches:
1. Remove these tags which are not rendering anyways from the tile, which keeps the bug in place, but shouldn't matter because no data consumer has taken notice of the issue (it would probably be easier to migrate to the new tag than to post a bug report!), _and they're going to be removed eventually anyways_. That's the approach in this PR.
This PR removes cases of `oneway=0` as this is an assumed default. It is not necessary to have this tag on every road or path and by removing it we can save space in the tiles by only including the meaningful values. It leaves intact `oneway=1` and `oneway=-1`, the latter of which indicates a oneway in the opposite direction of the way.
Unblocks #1265
This PR removes unnecessary `ramp` tagging from the `transportation` layer. The `ramp` tag is used for two situations. First, it is set for all `highway=***_link` tags to indicate a highway ramp. The second situation is when the `ramp` tag is set on a highway object, which indicates the presence of a ramp, such as one that would be used for a wheelchair. `ramp=0` is a reasonable default in these situations; by removing cases of `ramp=0` and presenting only `ramp=1` to the tiles, we can save considerable size in the tiles. Additionally, there appears to be a bug in which all objects tagged `highway=steps` are inexplicably tagged `ramp=1`.
The changes as follows:
1. Remove `ramp=0` where it appears in the tile.
2. `highway=steps` no longer assumes `ramp=1` in cases where `ramp=*` is not explicitly set.
In the current behavior, `ramp=0` is tagged only in at z13+ but suppressed in lower zooms:
This PR unifies `ramp=0` suppression between z13+ and z12-.
This PR adds the ability to create SQL tests that ensure that OSM data is properly imported and updated in the OpenMapTiles data schema. The tests work by injecting test OSM data and updates into the database and checking to ensure that the data is properly loaded into the database using standard SQL statements. With this framework in place, developers can now write small tests to inject known data into the database and ensure that imports and updates are working correctly.
In addition to the framework, basic tests are provided for four layers. These initial tests are in no way comprehensive, but they provide a structure and framework for developers to add their own tests or expand the existing ones to cover more cases.
Usage:
`make clean && make sql-test`
## How it works
The SQL tests consist of the following parts:
1. **Test import data**, located in `tests/import`. This test data is in the [OSM XML](https://wiki.openstreetmap.org/wiki/OSM_XML) format and contains the data that should be initially injected into the database. The files are numbered in order to ensure that each test data file contains OSM id numbers that are distinct from the other files. For example, the file starting with `100` will use node ids from 100000-199999, way ids from 1000-1999, and relation ids from 100-199.
1. **Test update data**, located in `tests/update`. This test data is in the [osmChange XML](https://wiki.openstreetmap.org/wiki/OsmChange) format, and contains the data that will be used to update the test import data (in order to verify that the update process is working correctly. These files are also numbered using the same scheme as the test import data.
1. **Import SQL test script**, located at `tests/test-post-import.sql`. This script is executed after the test import data has been injected, and runs SQL-based checks to ensure that the import data was properly imported. If there are failures in the tests, an entry will be added to the table `omt_test_failures`, with one record per error that occurs during the import process. A test failure will also fail the build. To inspect the test failure messages, run `make psql` and issue the comment `SELECT * FROM omt_test_failures`.
1. **Update SQL test script**, located at `tests/test-post-update.sql`. This script performs the same function as the import test script, except that it occurs after the test update data has been applied to the database. Note that script will only run if the import script passes all tests.
This PR does the following:
1. Suppresses underground buildings from rendering, when such buildings are tagged `location=underground`. Based on the wiki documentation, the `layer` tag does not determine whether a feature is above of below ground.
2. Unifies the building and building relation imposm exclusion mappings.
The following underground building feature in Luxembourg is used as the test object:
https://www.openstreetmap.org/way/582873794
* Add boundaries info to Readme
* make NE usage less specific - it may change later.
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
This PR fixes a bug in the `WHERE` clauses in the transportation layer generalization tables for zoom 10. The intended behavior of the layer is to suppress `highway=tertiary` and `highway=tertiary_link` at zoom 10 and lower. However, due to this bug, these objects were not suppressed as intended, because an `OR` was used in the SQL where an `AND` was needed instead.
This bug was inadvertently introduced in #1172😞
After commit b4b897999d the `make import-borders` rule no longer exists, but it's still mentioned in the README a couple of times. This PR removes those references.
Some [cleanup files are also mentioned in `.env`](4d6945b935/.env (L42-L44)). I'm not sure if those lines can be removed too, because the variables are used in [`bin/import-borders`](3796c4eab0/bin/import-borders (L8-L10)) in the `openmaptiles/openmaptiles-tools` repo.
Cache dir should always exist, so after deleting it,
it must be recreated. This is done in `make init-dirs`,
but if gets destroyed here, it won't auto-recreate.
This PR removes unneeded `CASE` clauses which converts `toll` tags to booleans. However, the `toll` tag is already mapped as a boolean by imposm, therefore this processing can be simplified.
I was reading through the SQL used to generalise the landcover layer ([`layers/landcover/generalized.sql`](596f44aa26/layers/landcover/generalized.sql)) and I noticed that when creating the generalised landcover tables for zooms 9-13, the SQL was doing something like this:
```sql
SELECT * FROM simplify_vw_z13 WHERE ST_NPoints(geometry) < 50
UNION ALL
SELECT * FROM simplify_vw_z13 WHERE ST_NPoints(geometry) >= 50 AND ST_NPoints(geometry) < 300
```
As in, the simplification/clustering of landcover generalisations was being done in two steps: once for polygons with fewer than fifty points, and once for polygons with 50+ points.
As far as I can see, there's no benefit to doing this — it's probably just an artefact of the dev work done for the [original pull request](https://github.com/openmaptiles/openmaptiles/pull/1035). Of course, I might be completely missing the reason for clustering polygons into two groups (<50 points and 50+ points). But assuming I haven't, I've created this pull request to simplify the SQL used in the generalisation, merging the two steps into something like this:
```sql
SELECT * FROM simplify_vw_z13 WHERE ST_NPoints(geometry) < 300
```
The effect is to slightly reduce the number of landcover features (polygons with <50 points can now be clustered/unioned with those 50+), and to reduce the time taken to generalise the landcover tables (in a small test using OSM's Iceland data I saw a 7% reduction in the time spent). It doesn't alter the features that are shown on the map.
Builds upon work from commits 4a1b0afa26 and da689f9e42.
Brian Sperlongano
