This PR updating Natural Earth dataset from v4.1 to 5.1.2. It is in cooperation with https://github.com/openmaptiles/openmaptiles-tools/pull/414, where is `import-data` docker image updated.
PR removes manual fixes for lake merging (already fixed in version 5.1.2)
Switched comparison from `gn_ascii` (removed from NE5) to `name_en`.
Could be merged after https://github.com/openmaptiles/openmaptiles-tools/pull/413. After OMT-T is merged, it will use `latest` image (until the release of OMT-T v7)
* Add info about using MID_ZOOM to optimize tile generation
* Remove params for generation that are now automatically detected
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
* Allow setting `BBOX` to be set in `.env` file
Currently, the `BBOX` setting in `.env` is ignored for all areas except `planet`.
On the other hand, the `planet` area is an overkill for any `BBOX` setting other than the default - `-180.0,-85.0511,180.0,85.0511`.
With this PR, `quickstart.sh` would not override a modified `BBOX` value in `.env`.
Also, this provides a way to avoid the pessimistic `BBOX` computation for `osmfr` extracts, as described in https://github.com/openmaptiles/openmaptiles-tools/pull/297#issuecomment-700792702
#### Currently
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env` and a `planet` data source is used, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file.
- If a user sets the `BBOX` value in `.env` and a non-`planet` data source is used, it is ignored - `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source.
#### Problem statement
While users of a `planet` data source have a simple way to override the default extend of the tile generation, users of other data sources have no simple way of doing that. In fact, for such users the `BBOX` setting in the `.env` file is ignored and therefore misleading.
#### Proposal
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env`, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file. This applies to `planet` and other data sources.
Move the hard coded `openmaptiles.yaml` from Makefile to a variable. Allows use other tileset definition.
A step forward easy alternative layer definition.
* Set `MAX_ZOOM` to 7 by default.
* Remove `QUICKSTART_MIN/MAX_ZOOM` - unneeded complexity with two env vars. We can just use `MIN_ZOOM` and `MAX_ZOOM`. See also #261
* Generate dc-config yaml file with a new `make generate-dc-config` step. It will compute BBOX based on the downloaded data file. This step is not needed for planet generation.
* Generate Imposm replication file only when `DIFF_MODE` is `true`. Not needed otherwise. If the data source does not support it, it will throw an error.
Closes#904
* Make all data-related targets like `download*`, `import-osm`, `import-borders`, and `generate-tiles` into `area`-aware -- making it possible for multiple data files to coexist inside the `./data` dir.
* Add `make download area=... [url=...]` command to automatically download any kind of area by checking Geofabrik, BBBike, and OSM.fr, optionally from a custom URL. Supports `area=planet` too.
* Do not re-download area with `make download-*` if it already exists.
* Automatically rename `<area>-latest.osm.pbf` into `<area>.osm.pbf`
* If `area=...` parameter is not given to `make`, see if there is exactly one `*.osm.pbf` file, and if so, use `*` as the `area`.
* Configure many variables in the .env file, overriding the defaults in tools
* If `<area>.osm.pbf` exists, but `<area>.dc-config.pbf` is missing, generate it using `download-osm make-dc` command.
Also:
* closes#614
* closes#647
* partially addresses #261
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
This is a partial migration of https://github.com/openmaptiles/openmaptiles/pull/785
* Use `import-data` instead of `import-lakelines`, `import-water`, and `import-natural-earth`
* Upgrade docker-compose.yml to version 2.3 (allows some extra env var usage in yaml file itself)
* Remove `openmaptiles-tools:latest` usage -- no longer needed, can use current version 4.1
* `db-start` does not do a container recreation in case docker-compose.yml definition has changed.
* a few minor cleanups in quickstart.sh
import-wikidata should run before import-sql, and v4.1 allows for that.
Also it optimizes the wd_names table to just be a simple Wikidata ID -> labels lookup, with a proper index.
Minor other changes:
* `test-perf-null` target is now part of the Makefile
* `./data/osmstat.txt` is no longer created
* area download file now in this format: `${osm_area}-latest.osm.pbf`
* Switch OMT to use the new tools v4.0.0
* borders are dynamically generated from the PBF file instead of downloading a prepared CSV file
* all tools are executed as current user instead of root, thus files are easier to modify/delete if needed
* all data is stored in the local file system instead of docker volumes (Docker currently has a limitation of non-root operation for internal volumes). This also makes it easier to examine and test it.
* New `init-dirs` make target creates all the needed dirs - `build, data, cache`
* `make clean` deletes the whole `build` dir instead of individual files.
* `clean-docker` for backward compatibility deletes `cache` dirs (it used to be a volume)
* all `psql` calls are now done with `ON_ERROR_STOP=1`
* got rid of `pgclimb-*` targets -- same results can be done with `psql` (`pgclimb-list-views` & `pgclimb-list-tables` renamed to `list-views` and `list-tables`)
* rail stations
* Create 'railway' POI class with 'station' and 'halt' subclasses
* Introduce subway stops as POI
* Revert unwanted changes in .env and openmaptiles.yaml
* Remove halt key from documentation
* Distinguish aeroway and railway station, add tram_stop
Tomas Pohanka