* OpenMapTiles style
* Rename style snippets from layer_name.json to defaultstyle.json
* Add README to icons dir.
* Add README to icons dir.
* Rename snippets to style.json
* Fix spritezero - move icons to svg subdir
* Move icons from style/icons/svg directly to style/icons
Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
Since PR #922 the contents of `.env` are included in `Makefile` in order for `make` to by aligned with the `docker-compose` settings.
```
# Make all .env variables available for make targets
include .env
```
The down-side of employing the `include` mechanism is that the settings in `.env` now take higher precedence than the shell environment variables. As a result, controlling the OpenMapTiles flow because more difficult. For example, tests for `DIFF_MODE=true` had to modify the contents of `.env` in order to work:
b0e7f7884c/.github/workflows/integrity.yml (L45-L47) and
b0e7f7884c/Makefile (L629-L630)
Users were also faced with similar difficulties.
This PR restores the ability to control `make` and `quickstart.sh` using environment variable while keeping the use of the `.env` at a lower priority.
The result is restoring the ability to easily adjust the OpenMapTiles flow using environment variables, such as:
```
PGPORT=54321 DIFF_MODE=true ./quickstart.sh monaco
```
#### Notes:
1. This PR depends on #1363
2. This PR includes some clean-up of `Makefile`
Make targets `start-tileserver`, `start-maputnik`, and `update-osm` initialize on-going processes rather than perform a one-time task. It is more appropriate to execute them using `docker-compose up -d`. With the current `docker-compose run` execution, the user needs to run `make` as a background process. This PR also allows the use of `docker-compose logs` rather than the need to handle `stdout` and `stderr` redirection and avoids the dependency of `stdin` handling for `docker-compose run`
Using `docker-compose` rather than `docker`
Use appropriate` start-*` and `stop-*` make target names
Resolves#1352
Remove `FILTER_MAPNIK_OUTPUT` variables from docker-compose which stop raising warnings during start-up dockers.
This functionality was removed in https://github.com/openmaptiles/openmaptiles-tools/pull/349.
The current version of the mapnik does not need `FILTER_MAPNIK_OUTPUT`.
Fixes#1156Fixes#810Fixes#1228
This PR replaces `osmborder`, which is no longer maintained, with `imposm` mappings and SQL code to generate borders. Key features that were moved into the imposm/SQL layer:
1. Grouping by `osm_id` and aggregating by lowest `admin_level` value so that there's only one copy of ways that are members of multiple relations.
2. Filtering out of point features in boundary relations (typically `admin_centre` and `label` roles).
3. Move disputed boundary detection logic into SQL.
This will increase the database size slightly because of the limits of what imposm can do, as some of the filtering is done in the SQL layer after importing, rather than being done in `osmborder`.
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
The current option of postserve in the docker-compose produce an invalid tilejson.
```json
"tiles": ["localhost:8090/tiles/{z}/{x}/{y}.pbf"]
```
`tiles` should be URL. The protocol is missing.
Not sure about the best way to fix this. Just adding `http://` to the docker-compose.yaml does not allow https usage. But setting protocol in `OMT_HOST` seems weird.
Could add an extra OMT_PROTOCOL with `http` as default.
Allow MIN_ZOOM, and MAX_ZOOM to be overwritten from shell for `quickstart.sh`, `make generate-dc-config`, and other `docker-compose openmaptiles-tools` uses.
For example,
```
export MAX_ZOOM=14; ./quickstart.sh monaco
```
ignores the `MAX_ZOOM=14` environment variable and uses the `MAX_ZOOM=7` value from `.env` file
This PR will also make this code unecessary
5c640daf4f/.github/workflows/tests.yml (L26-L28)
Move the hard coded `openmaptiles.yaml` from Makefile to a variable. Allows use other tileset definition.
A step forward easy alternative layer definition.
* Current code incorrectly passes `IMPOSM_CONFIG_FILE` to the `generate-tiles` image, but should pass it to the tools.
* add a test to ensure imposm config exists
* add a test to ensure area is set during updates
Closes#904
* Make all data-related targets like `download*`, `import-osm`, `import-borders`, and `generate-tiles` into `area`-aware -- making it possible for multiple data files to coexist inside the `./data` dir.
* Add `make download area=... [url=...]` command to automatically download any kind of area by checking Geofabrik, BBBike, and OSM.fr, optionally from a custom URL. Supports `area=planet` too.
* Do not re-download area with `make download-*` if it already exists.
* Automatically rename `<area>-latest.osm.pbf` into `<area>.osm.pbf`
* If `area=...` parameter is not given to `make`, see if there is exactly one `*.osm.pbf` file, and if so, use `*` as the `area`.
* Configure many variables in the .env file, overriding the defaults in tools
* If `<area>.osm.pbf` exists, but `<area>.dc-config.pbf` is missing, generate it using `download-osm make-dc` command.
Also:
* closes#614
* closes#647
* partially addresses #261
* `make start-postserve PPORT=8000`
* `make start-tileserver TPORT=9000`
Note that both ports need to be the same on host and inside the docker container because the services include a URL to the tiles as part of their manifest.
Generate multiple SQL files to be imported in parallel.
The files will respect the cross-layer dependencies,
so they can be all ingested at the same time.
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
* allow postgres image to be overwritten with an env var
* allow DIFF_MODE var to be overwritten with an env var
* add /mapping and /cache dirs into tools image
* make `build-sql` target explicit rather than relying on a filename
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* better output messages
This is a partial migration of https://github.com/openmaptiles/openmaptiles/pull/785
* Use `import-data` instead of `import-lakelines`, `import-water`, and `import-natural-earth`
* Upgrade docker-compose.yml to version 2.3 (allows some extra env var usage in yaml file itself)
* Remove `openmaptiles-tools:latest` usage -- no longer needed, can use current version 4.1
* `db-start` does not do a container recreation in case docker-compose.yml definition has changed.
* a few minor cleanups in quickstart.sh
* Switch OMT to use the new tools v4.0.0
* borders are dynamically generated from the PBF file instead of downloading a prepared CSV file
* all tools are executed as current user instead of root, thus files are easier to modify/delete if needed
* all data is stored in the local file system instead of docker volumes (Docker currently has a limitation of non-root operation for internal volumes). This also makes it easier to examine and test it.
* New `init-dirs` make target creates all the needed dirs - `build, data, cache`
* `make clean` deletes the whole `build` dir instead of individual files.
* `clean-docker` for backward compatibility deletes `cache` dirs (it used to be a volume)
* all `psql` calls are now done with `ON_ERROR_STOP=1`
* got rid of `pgclimb-*` targets -- same results can be done with `psql` (`pgclimb-list-views` & `pgclimb-list-tables` renamed to `list-views` and `list-tables`)
* Use _resolve_wikidata in layer mapping.yaml
Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.
This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.
The next tools version will remove it when generating imposm mapping file.
* Migrate to new Wikidata importer
Uses latest tools to populate the wd_names table
during the quickstart. This can be merged already,
or we can wait for the next tools version.
* Use unified tools version for all images
* do not start postserve as part of quickstart, but added a help message how to start it
* wait for SQL start with pgwait
Thomas Helmrich