When `make clean-unnecessary-docker` is run twice, or when there are no docker images tagges as "<none>", make fails because grep fails:
```
Deleting unnecessary container(s)...
Deleting unnecessary image(s)...
make: *** [Makefile:562: clean-unnecessary-docker] Error 1
```
Using `awk` to _also_ search for "<none>", instead of `grep`, solves this issue.
On creation of materialized views `osm_transportation_merge_linestring_gen_zX`, an index is created each time the new view and not used else where than the next generalization step.
It does not worth it to to a scan on the table to create index that will be used only once, while the next generalisation step can also be done in one full scan.
So, removing these indices.
This adds `place=quarter` which was introduced in 2011 as a subdivision
of villages/towns/cities as a value that sits between `place=suburb` and
`place=neighbourhood`. It has by now been used over 40,000 times and is
rendered in the OSM-Carto.
Add missing `NOT` in WHERE condition.
It causes there are duplicated features of subclass `wood` and `forest` and features of other subclasses are missing.
Bug introduced in #944.
Missing exclamation mark in not equal operator caused that tags were not updated thus there were missing `name_int`, `name:latin` and `name:nonlatin`.
This PR solving the speed and memory issue with buildings aggregation on zoom level 13. The problem was to create ST_ClusterDBScan which failed after try to cluster a larger area. Now it will make cluster only in "small" polygon using 'country_osm_grid` which covers the world's lands.
This PR solving #1022 and #974
During the import-osm step, there are three generalized tables for the waterway that are not used for subsequent processing.
Thanks @lazaa32 for the notice.
This PR brings few modification which make it possible to generalize landcover on larger areas. Tested on Europe.
- Use ST_SnapToGrid to avoid ERROR: GEOSUnaryUnion: TopologyException: found non-noded intersection between LINESTRING.
- Cluster only landcover subclasses: wood and forest.
- Use less generalized tables as a starting point for next step generalization.
This PR adds state name to national borders (`admin_level=2`)
From zoom level 3 there is an added `adm0_l` and `adm0_r` attribute for national borders.
There is 3 main steps:
1) union border lines from OSM and keep separate LineString between intersects
2) create polygons from step 1), create a point in each state polygon, add to this point a state abbreviation from NaturalEarth Data
3) create a short parallel line on the left and right of a small part of the border in 70m distance, and add information about which state overlap this short line.
With the new release of OMT-T (5.3) are available tools [`layer-stat`](https://github.com/openmaptiles/openmaptiles-tools/pull/293).
With the new release of OMT-T can be replaced `make` target `generate-qareports` by `generate-qa`
Used as:
```
make generate-qa STAT_FUNCTION=frequency LAYER=transportation ATTRIBUTE=class
```
Instead of using [Douglas-Peucker algorithm](https://postgis.net/docs/ST_SimplifyPreserveTopology.html) which is using for generalized tables [imposm](https://imposm.org/docs/imposm3/latest/mapping.html#generalized-tables) it is used [Visvalingam-Whyatt algorithm](https://postgis.net/docs/ST_SimplifyVW.html).
Solution:
remove imposm generalized tables (during `import-osm`) and create generalization in `import-sql` step for zooms 7-14. Zooms 0-6 are from Natural Earth data.
Upper zoom levels (7 and 8) are ok to merge with no big impact on creation speed. In Canada from z9 it took too long. Between zoom levels 10 - 13 there can be union polygons with less than 300 edge points (empirical number based on a test on Canada forest, can be discussed). Polygons with more than 300 edge points are just simplified. Zoom 14 is from the original dataset `osm_landcover_polygon`.
There is also removed `osm_id` which is never used and not passed into vector tiles.
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
* Allow setting `BBOX` to be set in `.env` file
Currently, the `BBOX` setting in `.env` is ignored for all areas except `planet`.
On the other hand, the `planet` area is an overkill for any `BBOX` setting other than the default - `-180.0,-85.0511,180.0,85.0511`.
With this PR, `quickstart.sh` would not override a modified `BBOX` value in `.env`.
Also, this provides a way to avoid the pessimistic `BBOX` computation for `osmfr` extracts, as described in https://github.com/openmaptiles/openmaptiles-tools/pull/297#issuecomment-700792702
#### Currently
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env` and a `planet` data source is used, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file.
- If a user sets the `BBOX` value in `.env` and a non-`planet` data source is used, it is ignored - `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source.
#### Problem statement
While users of a `planet` data source have a simple way to override the default extend of the tile generation, users of other data sources have no simple way of doing that. In fact, for such users the `BBOX` setting in the `.env` file is ignored and therefore misleading.
#### Proposal
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env`, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file. This applies to `planet` and other data sources.
https://github.com/openmaptiles/openmaptiles/issues/999
- Point 1 solved only partially - there is `brunnel` value added into zooms 9, 10, 11. For zooms 8 and smaller it would need change in osm_transportation_merge_ tables - I am going to try it but maybe its SQL costs will be to high.
- Point 2 solved.
The ocean layer (table osm_ocean_polygon insert by `make import-data`) is made from squares that have a 10m buffer.
This creates extra geometry in vector tiles as:
Union of full ocean squares should decrease the size of the water layer.
there are 8042 polygons (squares, 5 points, more then 100km<sup>2</sup>), which was reduced to 22 polygons - 1184kB vs 40kB respectively.
In mbtiles should be the size reduction even more significant, due to the creation of more polygons based on a 10m buffer of each square.
Korean moved from `ko_rm` to `ko-Latn`. Same as Japanese deprecated `ja_rm` in advance of `ja-Latn`, but still have twice more records. For Japanese add Hiragana form.
fix#537
listed in #930
The current option of postserve in the docker-compose produce an invalid tilejson.
```json
"tiles": ["localhost:8090/tiles/{z}/{x}/{y}.pbf"]
```
`tiles` should be URL. The protocol is missing.
Not sure about the best way to fix this. Just adding `http://` to the docker-compose.yaml does not allow https usage. But setting protocol in `OMT_HOST` seems weird.
Could add an extra OMT_PROTOCOL with `http` as default.
- Use `find` instead of `wildcard` to detect existing `.osm.pbf` files
- Use `patsubst` to extract an `area` from a filename
- Clarify messssage, especially for errors
Resolve https://github.com/openmaptiles/openmaptiles/issues/1008
The index `osm_highway_linestring_highway_idx` is not used. There is also a used partial index `osm_highway_linestring_highway_partial_idx`.
I cross check the usage of the index from postgres query stats and all SQL query code where the table `osm_highway_linestring` is refered.
This index is relatively big.
Add salt pond as water.
Most of salt pond are already tagged with natural=water even if it's not required. So the load added by this tag is very light.
Current implementation of osm_important_waterway_linestring have two bugs:
* The distinct on is_old keep the oldest version of the old object, and the oldest version of the new object, but need the last version of the new object.
* Delete the old version of the object and rebuild the using the new version of the object. But we need to remove matchings the old and the new version of the object, then rebuild the two.
Then only delete and update using first and last version of the object, intermediate versions are ignored.
Similar implementation of what is done in #996.
Replacing materialized view by a tables with update from trigger on change only.
Differential update of `osm_transportation_name_linestring`, `osm_transportation_name_linestring` and `osm_transportation_name_linestring_genX`).
At the end of the transaction the dependent rows are updated.
The goal is to update more quickly the content of derivated tables by just updating the changing content. It replaces the update of materialized view because their need a full recompute (with lock issue).
It is the last part of the replacement of materialized view for the transportation layer.
It addresses #814 and a part of #809.
Rémy Léone