When `make clean-unnecessary-docker` is run twice, or when there are no docker images tagges as "<none>", make fails because grep fails:
```
Deleting unnecessary container(s)...
Deleting unnecessary image(s)...
make: *** [Makefile:562: clean-unnecessary-docker] Error 1
```
Using `awk` to _also_ search for "<none>", instead of `grep`, solves this issue.
With the new release of OMT-T (5.3) are available tools [`layer-stat`](https://github.com/openmaptiles/openmaptiles-tools/pull/293).
With the new release of OMT-T can be replaced `make` target `generate-qareports` by `generate-qa`
Used as:
```
make generate-qa STAT_FUNCTION=frequency LAYER=transportation ATTRIBUTE=class
```
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
- Use `find` instead of `wildcard` to detect existing `.osm.pbf` files
- Use `patsubst` to extract an `area` from a filename
- Clarify messssage, especially for errors
Resolve https://github.com/openmaptiles/openmaptiles/issues/1008
Move the hard coded `openmaptiles.yaml` from Makefile to a variable. Allows use other tileset definition.
A step forward easy alternative layer definition.
* Current code incorrectly passes `IMPOSM_CONFIG_FILE` to the `generate-tiles` image, but should pass it to the tools.
* add a test to ensure imposm config exists
* add a test to ensure area is set during updates
Make sure that core targets (i.e. part of the `all` target like build-sql and build yaml files) do not run multiple times if they already exist.
Makefile for some reason does not like it when a real target depends on a PHONY target, and re-runs it. I added an `if` statement to skip building targets if their result already exists.
* Set `MAX_ZOOM` to 7 by default.
* Remove `QUICKSTART_MIN/MAX_ZOOM` - unneeded complexity with two env vars. We can just use `MIN_ZOOM` and `MAX_ZOOM`. See also #261
* Generate dc-config yaml file with a new `make generate-dc-config` step. It will compute BBOX based on the downloaded data file. This step is not needed for planet generation.
* Generate Imposm replication file only when `DIFF_MODE` is `true`. Not needed otherwise. If the data source does not support it, it will throw an error.
Closes#904
* Make all data-related targets like `download*`, `import-osm`, `import-borders`, and `generate-tiles` into `area`-aware -- making it possible for multiple data files to coexist inside the `./data` dir.
* Add `make download area=... [url=...]` command to automatically download any kind of area by checking Geofabrik, BBBike, and OSM.fr, optionally from a custom URL. Supports `area=planet` too.
* Do not re-download area with `make download-*` if it already exists.
* Automatically rename `<area>-latest.osm.pbf` into `<area>.osm.pbf`
* If `area=...` parameter is not given to `make`, see if there is exactly one `*.osm.pbf` file, and if so, use `*` as the `area`.
* Configure many variables in the .env file, overriding the defaults in tools
* If `<area>.osm.pbf` exists, but `<area>.dc-config.pbf` is missing, generate it using `download-osm make-dc` command.
Also:
* closes#614
* closes#647
* partially addresses #261
* Make a few spacing adjustments for ease-of-reading and consistency
* fix bbbike naming and other source urls
* remove unneeded `override`
* added `list-bbbike` target
This only changes the generated `getmvt(...)` function used by mvt users.
Beginning with tools 5.1, postgis docker images uses postgis v3.0, and includes gzip extension
* `make start-postserve PPORT=8000`
* `make start-tileserver TPORT=9000`
Note that both ports need to be the same on host and inside the docker container because the services include a URL to the tiles as part of their manifest.
Add `getmvt` sql function as part of the build-sql make step. The function will be imported into PG during the `import-sql` step after all other SQL is done. This is a noop for the `generate-tiles` approach.
* make sure all required dirs are initialized before running any of the docker commands. Otherwise docker will create them as root.
* minor docs cleanup
Generate multiple SQL files to be imported in parallel.
The files will respect the cross-layer dependencies,
so they can be all ingested at the same time.
* Adds a script to downloads multiple areas and compute their test parameters
* added a large test that uses a combined 76MB file with equatorial-guinea, liechtenstein, district-of-columbia, greater-london
* cache wikidata downloads
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
* allow postgres image to be overwritten with an env var
* allow DIFF_MODE var to be overwritten with an env var
* add /mapping and /cache dirs into tools image
* make `build-sql` target explicit rather than relying on a filename
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* better output messages
This is a partial migration of https://github.com/openmaptiles/openmaptiles/pull/785
* Use `import-data` instead of `import-lakelines`, `import-water`, and `import-natural-earth`
* Upgrade docker-compose.yml to version 2.3 (allows some extra env var usage in yaml file itself)
* Remove `openmaptiles-tools:latest` usage -- no longer needed, can use current version 4.1
* `db-start` does not do a container recreation in case docker-compose.yml definition has changed.
* a few minor cleanups in quickstart.sh
Yuri Astrakhan