Cache dir should always exist, so after deleting it,
it must be recreated. This is done in `make init-dirs`,
but if gets destroyed here, it won't auto-recreate.
Avoid redirection of error messages to output files
- build/openmaptiles.tm2source/data.yml
- build/mapping.yaml
Currently, if an error occurs when `make all` creates any of these files, the error message is not printed to the console.
```
docker-compose run --rm --user=1000:1000 openmaptiles-tools generate-tm2source openmaptiles.yaml --host="postgres" --port=5432 --database="openmaptiles" --user="openmaptiles" --password="openmaptiles" > build/openmaptiles.tm2source/data.yml
Creating openmaptiles_openmaptiles-tools_run ... done
ERROR: 1
make: *** [Makefile:257: build/openmaptiles.tm2source/data.yml] Error 1
```
An inexperienced user would not know where to look for the details.
With this PR, the error is sent to the console:
```
docker-compose run --rm --user=1000:1000 openmaptiles-tools bash -c \
'generate-tm2source openmaptiles.yaml --host="postgres" --port=5432 --database="openmaptiles" --user="openmaptiles" --password="openmaptiles" > build/openmaptiles.tm2source/data.yml'
Creating openmaptiles_openmaptiles-tools_run ... done
Could not parse /tileset/layers/aeroway/mapping.yaml
found undefined alias 'refkjbkjkhh'
in "/tileset/layers/aeroway/mapping.yaml", line 94, column 7
ERROR: 1
make: *** [Makefile:257: build/openmaptiles.tm2source/data.yml] Error 1
```
Redirection of stdout is now done in the container, leaving stderr to go to the host.
Fixes#1156Fixes#810Fixes#1228
This PR replaces `osmborder`, which is no longer maintained, with `imposm` mappings and SQL code to generate borders. Key features that were moved into the imposm/SQL layer:
1. Grouping by `osm_id` and aggregating by lowest `admin_level` value so that there's only one copy of ways that are members of multiple relations.
2. Filtering out of point features in boundary relations (typically `admin_centre` and `label` roles).
3. Move disputed boundary detection logic into SQL.
This will increase the database size slightly because of the limits of what imposm can do, as some of the filtering is done in the SQL layer after importing, rather than being done in `osmborder`.
Because of the parallel processing of `import-sql` the error message is printed in the middle of the output. With this PR the error is displayed again at the end:
Issue: https://github.com/openmaptiles/openmaptiles-tools/pull/370
```
awk '1{print; fflush()} $$0~".*ERROR" {txt=$$0} END{ if(txt){print "\n*** ERROR detected, aborting:"; print txt; exit(1)} }'
```
Explanation:
- `1{print; fflush()}` means if true then print output -> `1{}` same as `if(true){}`
- `$$0~".*ERROR" {txt=$$0}` get first argument `$$0` (line of output) and check if it contains (regex string) `ERROR`. If true save line to var `txt`: `{txt=$$0}`
- `END{ ... }` if last line of output is reached
- `if(txt){print "\n*** ERROR detected, aborting:"; print txt; exit(1)}` if error was found in a line / var `txt` is existing print it out
Adding -T fixes the issue when users are unable to stop tile genartion with ctrl+c
For some reason it doesn't work without it, even though the process of building
and running both docker images is the same.
While importing multiple .pbf files into postgres I was confronted with some unnecessary problems:
1. I wanted to use the borders of the planet which I had already generated. But `make import-borders` needs a .pbf file which is not necessary if the line.csv file is already existing -> So I added a check if the line.csv is **not** existing it checks for the `area parameter` / .pbf file
2. `make generate-tiles-pg` is only possible with a specified `area parameter` BUT the area is not used / needed for generation -> So I removed the area check
This PR makes three improvements to the current `make help` output:
1. Adds additional help entries for useful `make` targets that were not currently in `make help`
2. Pipes the help output through `less` so that the user can scroll up or down through the various `make` targets if it overflows the length of the user's screen
3. Re-groups the "developer" targets into separate entries for database and docker-related commands
In addition, an unneeded whitespace was removed from line 7 of the Makefile.
When `make clean-unnecessary-docker` is run twice, or when there are no docker images tagges as "<none>", make fails because grep fails:
```
Deleting unnecessary container(s)...
Deleting unnecessary image(s)...
make: *** [Makefile:562: clean-unnecessary-docker] Error 1
```
Using `awk` to _also_ search for "<none>", instead of `grep`, solves this issue.
With the new release of OMT-T (5.3) are available tools [`layer-stat`](https://github.com/openmaptiles/openmaptiles-tools/pull/293).
With the new release of OMT-T can be replaced `make` target `generate-qareports` by `generate-qa`
Used as:
```
make generate-qa STAT_FUNCTION=frequency LAYER=transportation ATTRIBUTE=class
```
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
- Use `find` instead of `wildcard` to detect existing `.osm.pbf` files
- Use `patsubst` to extract an `area` from a filename
- Clarify messssage, especially for errors
Resolve https://github.com/openmaptiles/openmaptiles/issues/1008
Move the hard coded `openmaptiles.yaml` from Makefile to a variable. Allows use other tileset definition.
A step forward easy alternative layer definition.
* Current code incorrectly passes `IMPOSM_CONFIG_FILE` to the `generate-tiles` image, but should pass it to the tools.
* add a test to ensure imposm config exists
* add a test to ensure area is set during updates
Make sure that core targets (i.e. part of the `all` target like build-sql and build yaml files) do not run multiple times if they already exist.
Makefile for some reason does not like it when a real target depends on a PHONY target, and re-runs it. I added an `if` statement to skip building targets if their result already exists.
* Set `MAX_ZOOM` to 7 by default.
* Remove `QUICKSTART_MIN/MAX_ZOOM` - unneeded complexity with two env vars. We can just use `MIN_ZOOM` and `MAX_ZOOM`. See also #261
* Generate dc-config yaml file with a new `make generate-dc-config` step. It will compute BBOX based on the downloaded data file. This step is not needed for planet generation.
* Generate Imposm replication file only when `DIFF_MODE` is `true`. Not needed otherwise. If the data source does not support it, it will throw an error.
Closes#904
* Make all data-related targets like `download*`, `import-osm`, `import-borders`, and `generate-tiles` into `area`-aware -- making it possible for multiple data files to coexist inside the `./data` dir.
* Add `make download area=... [url=...]` command to automatically download any kind of area by checking Geofabrik, BBBike, and OSM.fr, optionally from a custom URL. Supports `area=planet` too.
* Do not re-download area with `make download-*` if it already exists.
* Automatically rename `<area>-latest.osm.pbf` into `<area>.osm.pbf`
* If `area=...` parameter is not given to `make`, see if there is exactly one `*.osm.pbf` file, and if so, use `*` as the `area`.
* Configure many variables in the .env file, overriding the defaults in tools
* If `<area>.osm.pbf` exists, but `<area>.dc-config.pbf` is missing, generate it using `download-osm make-dc` command.
Also:
* closes#614
* closes#647
* partially addresses #261
* Make a few spacing adjustments for ease-of-reading and consistency
* fix bbbike naming and other source urls
* remove unneeded `override`
* added `list-bbbike` target
This only changes the generated `getmvt(...)` function used by mvt users.
Beginning with tools 5.1, postgis docker images uses postgis v3.0, and includes gzip extension
* `make start-postserve PPORT=8000`
* `make start-tileserver TPORT=9000`
Note that both ports need to be the same on host and inside the docker container because the services include a URL to the tiles as part of their manifest.
Add `getmvt` sql function as part of the build-sql make step. The function will be imported into PG during the `import-sql` step after all other SQL is done. This is a noop for the `generate-tiles` approach.
* make sure all required dirs are initialized before running any of the docker commands. Otherwise docker will create them as root.
* minor docs cleanup
Generate multiple SQL files to be imported in parallel.
The files will respect the cross-layer dependencies,
so they can be all ingested at the same time.
* Adds a script to downloads multiple areas and compute their test parameters
* added a large test that uses a combined 76MB file with equatorial-guinea, liechtenstein, district-of-columbia, greater-london
* cache wikidata downloads
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
Yuri Astrakhan
