Add missing `NOT` in WHERE condition.
It causes there are duplicated features of subclass `wood` and `forest` and features of other subclasses are missing.
Bug introduced in #944.
Missing exclamation mark in not equal operator caused that tags were not updated thus there were missing `name_int`, `name:latin` and `name:nonlatin`.
This PR solving the speed and memory issue with buildings aggregation on zoom level 13. The problem was to create ST_ClusterDBScan which failed after try to cluster a larger area. Now it will make cluster only in "small" polygon using 'country_osm_grid` which covers the world's lands.
This PR solving #1022 and #974
During the import-osm step, there are three generalized tables for the waterway that are not used for subsequent processing.
Thanks @lazaa32 for the notice.
This PR brings few modification which make it possible to generalize landcover on larger areas. Tested on Europe.
- Use ST_SnapToGrid to avoid ERROR: GEOSUnaryUnion: TopologyException: found non-noded intersection between LINESTRING.
- Cluster only landcover subclasses: wood and forest.
- Use less generalized tables as a starting point for next step generalization.
This PR adds state name to national borders (`admin_level=2`)
From zoom level 3 there is an added `adm0_l` and `adm0_r` attribute for national borders.
There is 3 main steps:
1) union border lines from OSM and keep separate LineString between intersects
2) create polygons from step 1), create a point in each state polygon, add to this point a state abbreviation from NaturalEarth Data
3) create a short parallel line on the left and right of a small part of the border in 70m distance, and add information about which state overlap this short line.
With the new release of OMT-T (5.3) are available tools [`layer-stat`](https://github.com/openmaptiles/openmaptiles-tools/pull/293).
With the new release of OMT-T can be replaced `make` target `generate-qareports` by `generate-qa`
Used as:
```
make generate-qa STAT_FUNCTION=frequency LAYER=transportation ATTRIBUTE=class
```
Instead of using [Douglas-Peucker algorithm](https://postgis.net/docs/ST_SimplifyPreserveTopology.html) which is using for generalized tables [imposm](https://imposm.org/docs/imposm3/latest/mapping.html#generalized-tables) it is used [Visvalingam-Whyatt algorithm](https://postgis.net/docs/ST_SimplifyVW.html).
Solution:
remove imposm generalized tables (during `import-osm`) and create generalization in `import-sql` step for zooms 7-14. Zooms 0-6 are from Natural Earth data.
Upper zoom levels (7 and 8) are ok to merge with no big impact on creation speed. In Canada from z9 it took too long. Between zoom levels 10 - 13 there can be union polygons with less than 300 edge points (empirical number based on a test on Canada forest, can be discussed). Polygons with more than 300 edge points are just simplified. Zoom 14 is from the original dataset `osm_landcover_polygon`.
There is also removed `osm_id` which is never used and not passed into vector tiles.
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
* Allow setting `BBOX` to be set in `.env` file
Currently, the `BBOX` setting in `.env` is ignored for all areas except `planet`.
On the other hand, the `planet` area is an overkill for any `BBOX` setting other than the default - `-180.0,-85.0511,180.0,85.0511`.
With this PR, `quickstart.sh` would not override a modified `BBOX` value in `.env`.
Also, this provides a way to avoid the pessimistic `BBOX` computation for `osmfr` extracts, as described in https://github.com/openmaptiles/openmaptiles-tools/pull/297#issuecomment-700792702
#### Currently
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env` and a `planet` data source is used, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file.
- If a user sets the `BBOX` value in `.env` and a non-`planet` data source is used, it is ignored - `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source.
#### Problem statement
While users of a `planet` data source have a simple way to override the default extend of the tile generation, users of other data sources have no simple way of doing that. In fact, for such users the `BBOX` setting in the `.env` file is ignored and therefore misleading.
#### Proposal
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env`, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file. This applies to `planet` and other data sources.
https://github.com/openmaptiles/openmaptiles/issues/999
- Point 1 solved only partially - there is `brunnel` value added into zooms 9, 10, 11. For zooms 8 and smaller it would need change in osm_transportation_merge_ tables - I am going to try it but maybe its SQL costs will be to high.
- Point 2 solved.
The ocean layer (table osm_ocean_polygon insert by `make import-data`) is made from squares that have a 10m buffer.
This creates extra geometry in vector tiles as:
Union of full ocean squares should decrease the size of the water layer.
there are 8042 polygons (squares, 5 points, more then 100km<sup>2</sup>), which was reduced to 22 polygons - 1184kB vs 40kB respectively.
In mbtiles should be the size reduction even more significant, due to the creation of more polygons based on a 10m buffer of each square.
Korean moved from `ko_rm` to `ko-Latn`. Same as Japanese deprecated `ja_rm` in advance of `ja-Latn`, but still have twice more records. For Japanese add Hiragana form.
fix#537
listed in #930
The current option of postserve in the docker-compose produce an invalid tilejson.
```json
"tiles": ["localhost:8090/tiles/{z}/{x}/{y}.pbf"]
```
`tiles` should be URL. The protocol is missing.
Not sure about the best way to fix this. Just adding `http://` to the docker-compose.yaml does not allow https usage. But setting protocol in `OMT_HOST` seems weird.
Could add an extra OMT_PROTOCOL with `http` as default.
- Use `find` instead of `wildcard` to detect existing `.osm.pbf` files
- Use `patsubst` to extract an `area` from a filename
- Clarify messssage, especially for errors
Resolve https://github.com/openmaptiles/openmaptiles/issues/1008
The index `osm_highway_linestring_highway_idx` is not used. There is also a used partial index `osm_highway_linestring_highway_partial_idx`.
I cross check the usage of the index from postgres query stats and all SQL query code where the table `osm_highway_linestring` is refered.
This index is relatively big.
Add salt pond as water.
Most of salt pond are already tagged with natural=water even if it's not required. So the load added by this tag is very light.
Current implementation of osm_important_waterway_linestring have two bugs:
* The distinct on is_old keep the oldest version of the old object, and the oldest version of the new object, but need the last version of the new object.
* Delete the old version of the object and rebuild the using the new version of the object. But we need to remove matchings the old and the new version of the object, then rebuild the two.
Then only delete and update using first and last version of the object, intermediate versions are ignored.
Similar implementation of what is done in #996.
Replacing materialized view by a tables with update from trigger on change only.
Differential update of `osm_transportation_name_linestring`, `osm_transportation_name_linestring` and `osm_transportation_name_linestring_genX`).
At the end of the transaction the dependent rows are updated.
The goal is to update more quickly the content of derivated tables by just updating the changing content. It replaces the update of materialized view because their need a full recompute (with lock issue).
It is the last part of the replacement of materialized view for the transportation layer.
It addresses #814 and a part of #809.
Allow MIN_ZOOM, and MAX_ZOOM to be overwritten from shell for `quickstart.sh`, `make generate-dc-config`, and other `docker-compose openmaptiles-tools` uses.
For example,
```
export MAX_ZOOM=14; ./quickstart.sh monaco
```
ignores the `MAX_ZOOM=14` environment variable and uses the `MAX_ZOOM=7` value from `.env` file
This PR will also make this code unecessary
5c640daf4f/.github/workflows/tests.yml (L26-L28)
When OSM roads in the `transportation_name` layer are stitched together, their grouping does not consider all `name:*` tags.
As a result, roads with different `name:*` tags may be stitched together.
The `waterway` layer performs the grouping properly, for the same purpose:
1685eaccbd/layers/waterway/update_important_waterway.sql (L34)
Co-authored-by: Frédéric Rodrigo <fred.rodrigo@gmail.com>
Reduce the size of the materialized view osm_transportation_name_network by moving the weare clause.
osm_transportation_name_network is only used for osm_transportation_name_linestring.
Co-authored-by: zstadler <zeev.stadler@gmail.com>
Replacing materialized view by a tables with update from trigger on change only.
The osm_id of object changing on table osm_route_member and osm_highway_linestring are logged.
At the end of the transaction the dependent row of osm_transportation_name_network are updated.
The goal is to update more quickly the content of derivated tables by just updating the changing content. It replaces the update of materialized view because their need a full recompute (with lock issue).
Note, it is only a part of the materialized view of the transportation layer, I am also working on the other parts, other parts are in #892 (omaterialized view on sm_transportation_name_linestring and osm_transportation_name_linestring_genX).
It addresses #814 and a part of #809.
Clean and do refactoring on `update_route_member.sql` as first step to replace materialized view osm_transportation_name_network by tables with diff update. See #892.
Adam Laža