* Chore(quickstart.sh):
Handle spaces in fonts name when calculating their md5 digest
* Chore(Makefile):
Change docker pull to docker-compose pull to read image from compose file
Some parts of quickstart.sh reference `${area}.mbtiles`, but the file that gets created with the default `.env` is `tiles.mbtiles`. We should either reference `$MBTILES_FILE` instead, or override the filename with `${area}.mbtiles` (which is the approach taken by this PR).
When docker compose returns a version with v prefix,
the version check would incorrectly treat it as version component
of 000, leading to always failing the docker-compose version check
even if the version is recent enough.
This happens at least with docker-compose 2.2.1 where
docker-compose version --short
returns v2.2.1 with the leading v.
Fixes#1156Fixes#810Fixes#1228
This PR replaces `osmborder`, which is no longer maintained, with `imposm` mappings and SQL code to generate borders. Key features that were moved into the imposm/SQL layer:
1. Grouping by `osm_id` and aggregating by lowest `admin_level` value so that there's only one copy of ways that are members of multiple relations.
2. Filtering out of point features in boundary relations (typically `admin_centre` and `label` roles).
3. Move disputed boundary detection logic into SQL.
This will increase the database size slightly because of the limits of what imposm can do, as some of the filtering is done in the SQL layer after importing, rather than being done in `osmborder`.
On some systems the `bc` command is not installed, so I added a check if the command can be used.
It is a simply calculation from KB to GB. If `bc` is missing, then it is displayed in KB.
fix: #1151
Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
* Allow setting `BBOX` to be set in `.env` file
Currently, the `BBOX` setting in `.env` is ignored for all areas except `planet`.
On the other hand, the `planet` area is an overkill for any `BBOX` setting other than the default - `-180.0,-85.0511,180.0,85.0511`.
With this PR, `quickstart.sh` would not override a modified `BBOX` value in `.env`.
Also, this provides a way to avoid the pessimistic `BBOX` computation for `osmfr` extracts, as described in https://github.com/openmaptiles/openmaptiles-tools/pull/297#issuecomment-700792702
#### Currently
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env` and a `planet` data source is used, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file.
- If a user sets the `BBOX` value in `.env` and a non-`planet` data source is used, it is ignored - `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source.
#### Problem statement
While users of a `planet` data source have a simple way to override the default extend of the tile generation, users of other data sources have no simple way of doing that. In fact, for such users the `BBOX` setting in the `.env` file is ignored and therefore misleading.
#### Proposal
- If the user does not do anything, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the full extent of the data source. This applies to `planet` and other data sources.
- If a user sets the `BBOX` value in `.env`, `quickstart.sh` and `make generate-tiles` create an `mbtiles` file for the extent set in the `.env` file. This applies to `planet` and other data sources.
* Set `MAX_ZOOM` to 7 by default.
* Remove `QUICKSTART_MIN/MAX_ZOOM` - unneeded complexity with two env vars. We can just use `MIN_ZOOM` and `MAX_ZOOM`. See also #261
* Generate dc-config yaml file with a new `make generate-dc-config` step. It will compute BBOX based on the downloaded data file. This step is not needed for planet generation.
* Generate Imposm replication file only when `DIFF_MODE` is `true`. Not needed otherwise. If the data source does not support it, it will throw an error.
Closes#904
* Make all data-related targets like `download*`, `import-osm`, `import-borders`, and `generate-tiles` into `area`-aware -- making it possible for multiple data files to coexist inside the `./data` dir.
* Add `make download area=... [url=...]` command to automatically download any kind of area by checking Geofabrik, BBBike, and OSM.fr, optionally from a custom URL. Supports `area=planet` too.
* Do not re-download area with `make download-*` if it already exists.
* Automatically rename `<area>-latest.osm.pbf` into `<area>.osm.pbf`
* If `area=...` parameter is not given to `make`, see if there is exactly one `*.osm.pbf` file, and if so, use `*` as the `area`.
* Configure many variables in the .env file, overriding the defaults in tools
* If `<area>.osm.pbf` exists, but `<area>.dc-config.pbf` is missing, generate it using `download-osm make-dc` command.
Also:
* closes#614
* closes#647
* partially addresses #261
* Make a few spacing adjustments for ease-of-reading and consistency
* fix bbbike naming and other source urls
* remove unneeded `override`
* added `list-bbbike` target
Generate multiple SQL files to be imported in parallel.
The files will respect the cross-layer dependencies,
so they can be all ingested at the same time.
* quickstart support for osmfr and bbike areas
- Use the `--empty` flag to start with an empty database
- Geofabrik as the default server
- osmfr is used for hierarchical area names such as `europe/austria`
- bbbike is used for Capitalized area names such as `Adelaide`
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
* allow postgres image to be overwritten with an env var
* allow DIFF_MODE var to be overwritten with an env var
* add /mapping and /cache dirs into tools image
* make `build-sql` target explicit rather than relying on a filename
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* better output messages
This is a partial migration of https://github.com/openmaptiles/openmaptiles/pull/785
* Use `import-data` instead of `import-lakelines`, `import-water`, and `import-natural-earth`
* Upgrade docker-compose.yml to version 2.3 (allows some extra env var usage in yaml file itself)
* Remove `openmaptiles-tools:latest` usage -- no longer needed, can use current version 4.1
* `db-start` does not do a container recreation in case docker-compose.yml definition has changed.
* a few minor cleanups in quickstart.sh
import-wikidata should run before import-sql, and v4.1 allows for that.
Also it optimizes the wd_names table to just be a simple Wikidata ID -> labels lookup, with a proper index.
Minor other changes:
* `test-perf-null` target is now part of the Makefile
* `./data/osmstat.txt` is no longer created
* area download file now in this format: `${osm_area}-latest.osm.pbf`
* Switch OMT to use the new tools v4.0.0
* borders are dynamically generated from the PBF file instead of downloading a prepared CSV file
* all tools are executed as current user instead of root, thus files are easier to modify/delete if needed
* all data is stored in the local file system instead of docker volumes (Docker currently has a limitation of non-root operation for internal volumes). This also makes it easier to examine and test it.
* New `init-dirs` make target creates all the needed dirs - `build, data, cache`
* `make clean` deletes the whole `build` dir instead of individual files.
* `clean-docker` for backward compatibility deletes `cache` dirs (it used to be a volume)
* all `psql` calls are now done with `ON_ERROR_STOP=1`
* got rid of `pgclimb-*` targets -- same results can be done with `psql` (`pgclimb-list-views` & `pgclimb-list-tables` renamed to `list-views` and `list-tables`)
* Use _resolve_wikidata in layer mapping.yaml
Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.
This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.
The next tools version will remove it when generating imposm mapping file.
* Migrate to new Wikidata importer
Uses latest tools to populate the wd_names table
during the quickstart. This can be merged already,
or we can wait for the next tools version.
* Use unified tools version for all images
* do not start postserve as part of quickstart, but added a help message how to start it
* wait for SQL start with pgwait
Yaser Kalali