Migrate to tools 5.3 and use the new bbox value.
* Use 5.3 - new bbox value
* Return correct bbox for meta-generate
mbtiles-tools meta-generate is now able to use bbox from *.bbox file
Thanks @nyurik
- Use `find` instead of `wildcard` to detect existing `.osm.pbf` files
- Use `patsubst` to extract an `area` from a filename
- Clarify messssage, especially for errors
Resolve https://github.com/openmaptiles/openmaptiles/issues/1008
Move the hard coded `openmaptiles.yaml` from Makefile to a variable. Allows use other tileset definition.
A step forward easy alternative layer definition.
* Current code incorrectly passes `IMPOSM_CONFIG_FILE` to the `generate-tiles` image, but should pass it to the tools.
* add a test to ensure imposm config exists
* add a test to ensure area is set during updates
Make sure that core targets (i.e. part of the `all` target like build-sql and build yaml files) do not run multiple times if they already exist.
Makefile for some reason does not like it when a real target depends on a PHONY target, and re-runs it. I added an `if` statement to skip building targets if their result already exists.
* Set `MAX_ZOOM` to 7 by default.
* Remove `QUICKSTART_MIN/MAX_ZOOM` - unneeded complexity with two env vars. We can just use `MIN_ZOOM` and `MAX_ZOOM`. See also #261
* Generate dc-config yaml file with a new `make generate-dc-config` step. It will compute BBOX based on the downloaded data file. This step is not needed for planet generation.
* Generate Imposm replication file only when `DIFF_MODE` is `true`. Not needed otherwise. If the data source does not support it, it will throw an error.
Closes#904
* Make all data-related targets like `download*`, `import-osm`, `import-borders`, and `generate-tiles` into `area`-aware -- making it possible for multiple data files to coexist inside the `./data` dir.
* Add `make download area=... [url=...]` command to automatically download any kind of area by checking Geofabrik, BBBike, and OSM.fr, optionally from a custom URL. Supports `area=planet` too.
* Do not re-download area with `make download-*` if it already exists.
* Automatically rename `<area>-latest.osm.pbf` into `<area>.osm.pbf`
* If `area=...` parameter is not given to `make`, see if there is exactly one `*.osm.pbf` file, and if so, use `*` as the `area`.
* Configure many variables in the .env file, overriding the defaults in tools
* If `<area>.osm.pbf` exists, but `<area>.dc-config.pbf` is missing, generate it using `download-osm make-dc` command.
Also:
* closes#614
* closes#647
* partially addresses #261
* Make a few spacing adjustments for ease-of-reading and consistency
* fix bbbike naming and other source urls
* remove unneeded `override`
* added `list-bbbike` target
This only changes the generated `getmvt(...)` function used by mvt users.
Beginning with tools 5.1, postgis docker images uses postgis v3.0, and includes gzip extension
* `make start-postserve PPORT=8000`
* `make start-tileserver TPORT=9000`
Note that both ports need to be the same on host and inside the docker container because the services include a URL to the tiles as part of their manifest.
Add `getmvt` sql function as part of the build-sql make step. The function will be imported into PG during the `import-sql` step after all other SQL is done. This is a noop for the `generate-tiles` approach.
* make sure all required dirs are initialized before running any of the docker commands. Otherwise docker will create them as root.
* minor docs cleanup
Generate multiple SQL files to be imported in parallel.
The files will respect the cross-layer dependencies,
so they can be all ingested at the same time.
* Adds a script to downloads multiple areas and compute their test parameters
* added a large test that uses a combined 76MB file with equatorial-guinea, liechtenstein, district-of-columbia, greater-london
* cache wikidata downloads
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
* allow postgres image to be overwritten with an env var
* allow DIFF_MODE var to be overwritten with an env var
* add /mapping and /cache dirs into tools image
* make `build-sql` target explicit rather than relying on a filename
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* better output messages
This is a partial migration of https://github.com/openmaptiles/openmaptiles/pull/785
* Use `import-data` instead of `import-lakelines`, `import-water`, and `import-natural-earth`
* Upgrade docker-compose.yml to version 2.3 (allows some extra env var usage in yaml file itself)
* Remove `openmaptiles-tools:latest` usage -- no longer needed, can use current version 4.1
* `db-start` does not do a container recreation in case docker-compose.yml definition has changed.
* a few minor cleanups in quickstart.sh
- Add `download-osmfr` and `download-bbbike` targets
- Port `DC_OPTS` to Windows
- Use make conditions instead of shell. Also simplifies `make -n` output
- Use remote Docker machine's IP, if defined, rather than `localhost` in `http://...` messages. Also applicable for Docker Toolbox for Windows.
- Align texts in `make help` output
- Update and bug-fix in `make remove-docker-images`
The implementation adds the `DC_PROJECT` parameter. It can also be set by an environment variable in the hosting shell. The environment variable can be overwritten by a make parameter, including `DC_PROJECT=` which restores the automatic project name.
#### _NOTE:_
It may be worthwhile to review the following make targets that are currently not impacted by the PR:
- `make start-postserve`
- `make list-docker-images`
- `make remove-docker-images`
- `make docker-unnecessary-clean`
This PR was suggested by @nyurik:
> zstadler 3:30 PM
Is it possible to run two instances of openmaptiles, and the postgis container in particular, on the same Linux host?
nyurik 4:17 PM
use docker-compose --project-name -- that should allow you to run everything in parallel
4:17
might need to update make file
4:17
btw, that would be a good PR for makefile -- to specify --project-name based on the current DIR name, but so that it can be overwritten by a makefile param
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
import-wikidata should run before import-sql, and v4.1 allows for that.
Also it optimizes the wd_names table to just be a simple Wikidata ID -> labels lookup, with a proper index.
Minor other changes:
* `test-perf-null` target is now part of the Makefile
* `./data/osmstat.txt` is no longer created
* area download file now in this format: `${osm_area}-latest.osm.pbf`
* Switch OMT to use the new tools v4.0.0
* borders are dynamically generated from the PBF file instead of downloading a prepared CSV file
* all tools are executed as current user instead of root, thus files are easier to modify/delete if needed
* all data is stored in the local file system instead of docker volumes (Docker currently has a limitation of non-root operation for internal volumes). This also makes it easier to examine and test it.
* New `init-dirs` make target creates all the needed dirs - `build, data, cache`
* `make clean` deletes the whole `build` dir instead of individual files.
* `clean-docker` for backward compatibility deletes `cache` dirs (it used to be a volume)
* all `psql` calls are now done with `ON_ERROR_STOP=1`
* got rid of `pgclimb-*` targets -- same results can be done with `psql` (`pgclimb-list-views` & `pgclimb-list-tables` renamed to `list-views` and `list-tables`)
Make a few more mappings declarative, and removes values declared in both SQL and the yaml file.
Here's the diff comparing `build/tileset.sql` in master vs the new PR. The changes are mostly stylistic, except when a nested `if` statement is expanded into individual `if ... and ...` conditions (logically identical)
```diff
55c55
diff --git a/build/tileset.sql b/build/tileset.sql
index 4e59357..7c6c444 100644
--- a/build/tileset.sql
+++ b/build/tileset.sql
@@ -52,7 +52,7 @@ CREATE INDEX IF NOT EXISTS osm_ocean_polygon_gen4_idx ON osm_ocean_polygon_gen4
CREATE OR REPLACE FUNCTION water_class(waterway TEXT) RETURNS TEXT AS $$
SELECT CASE
WHEN "waterway" IN ('', 'lake') THEN 'lake'
- WHEN "waterway"='dock' THEN 'dock'
+ WHEN "waterway" = 'dock' THEN 'dock'
ELSE 'river'
END;
$$ LANGUAGE SQL IMMUTABLE;
@@ -1813,24 +1813,41 @@ CREATE OR REPLACE FUNCTION highway_class(highway TEXT, public_transport TEXT, co
WHEN "highway" IN ('tertiary', 'tertiary_link') THEN 'tertiary'
WHEN "highway" IN ('unclassified', 'residential', 'living_street', 'road') THEN 'minor'
WHEN "highway" IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor')
- OR "public_transport"='platform'
+ OR "public_transport" = 'platform'
THEN 'path'
- WHEN "highway"='service' THEN 'service'
- WHEN "highway"='track' THEN 'track'
- WHEN "highway"='raceway' THEN 'raceway'
- WHEN highway = 'construction' THEN CASE
- WHEN construction IN ('motorway', 'motorway_link') THEN 'motorway_construction'
- WHEN construction IN ('trunk', 'trunk_link') THEN 'trunk_construction'
- WHEN construction IN ('primary', 'primary_link') THEN 'primary_construction'
- WHEN construction IN ('secondary', 'secondary_link') THEN 'secondary_construction'
- WHEN construction IN ('tertiary', 'tertiary_link') THEN 'tertiary_construction'
- WHEN construction IN ('', 'unclassified', 'residential', 'living_street', 'road') THEN 'minor_construction'
- WHEN construction IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor')
- OR public_transport = 'platform' THEN 'path_construction'
- WHEN construction = 'service' THEN 'service_construction'
- WHEN construction = 'track' THEN 'track_construction'
- WHEN construction = 'raceway' THEN 'raceway_construction'
- END
+ WHEN "highway" = 'service' THEN 'service'
+ WHEN "highway" = 'track' THEN 'track'
+ WHEN "highway" = 'raceway' THEN 'raceway'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('motorway', 'motorway_link')
+ THEN 'motorway_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('trunk', 'trunk_link')
+ THEN 'trunk_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('primary', 'primary_link')
+ THEN 'primary_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('secondary', 'secondary_link')
+ THEN 'secondary_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('tertiary', 'tertiary_link')
+ THEN 'tertiary_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('', 'unclassified', 'residential', 'living_street', 'road')
+ THEN 'minor_construction'
+ WHEN "highway" = 'construction'
+ AND ("construction" IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor') OR "public_transport" = 'platform')
+ THEN 'path_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" = 'service'
+ THEN 'service_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" = 'track'
+ THEN 'track_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" = 'raceway'
+ THEN 'raceway_construction'
END;
$$ LANGUAGE SQL IMMUTABLE;
@@ -4073,6 +4090,12 @@ RETURNS TEXT AS $$
WHEN "subclass" IN ('fast_food', 'food_court') THEN 'fast_food'
WHEN "subclass" IN ('park', 'bbq') THEN 'park'
WHEN "subclass" IN ('bus_stop', 'bus_station') THEN 'bus'
+ WHEN ("subclass" = 'station' AND "mapping_key" = 'railway')
+ OR "subclass" IN ('halt', 'tram_stop', 'subway')
+ THEN 'railway'
+ WHEN "subclass" = 'station'
+ AND "mapping_key" = 'aerialway'
+ THEN 'aerialway'
WHEN "subclass" IN ('subway_entrance', 'train_station_entrance') THEN 'entrance'
WHEN "subclass" IN ('camp_site', 'caravan_site') THEN 'campsite'
WHEN "subclass" IN ('laundry', 'dry_cleaning') THEN 'laundry'
@@ -4082,7 +4105,7 @@ RETURNS TEXT AS $$
WHEN "subclass" IN ('hotel', 'motel', 'bed_and_breakfast', 'guest_house', 'hostel', 'chalet', 'alpine_hut', 'dormitory') THEN 'lodging'
WHEN "subclass" IN ('chocolate', 'confectionery') THEN 'ice_cream'
WHEN "subclass" IN ('post_box', 'post_office') THEN 'post'
- WHEN "subclass"='cafe' THEN 'cafe'
+ WHEN "subclass" = 'cafe' THEN 'cafe'
WHEN "subclass" IN ('school', 'kindergarten') THEN 'school'
WHEN "subclass" IN ('alcohol', 'beverages', 'wine') THEN 'alcohol_shop'
WHEN "subclass" IN ('bar', 'nightclub') THEN 'bar'
@@ -4098,9 +4121,6 @@ RETURNS TEXT AS $$
WHEN "subclass" IN ('bag', 'clothes') THEN 'clothing_store'
WHEN "subclass" IN ('swimming_area', 'swimming') THEN 'swimming'
WHEN "subclass" IN ('castle', 'ruins') THEN 'castle'
- WHEN (subclass = 'station' AND mapping_key = 'railway')
- OR (subclass IN ('halt', 'tram_stop', 'subway')) THEN 'railway'
- WHEN (subclass = 'station' AND mapping_key = 'aerialway') THEN 'aerialway'
ELSE subclass
END;
$$ LANGUAGE SQL IMMUTABLE;
@@ -4301,22 +4321,22 @@ $$
COALESCE(NULLIF(name_de, ''), name, name_en) AS name_de,
tags,
CASE
- WHEN "aerodrome"='international'
- OR "aerodrome_type"='international'
+ WHEN "aerodrome" = 'international'
+ OR "aerodrome_type" = 'international'
THEN 'international'
- WHEN "aerodrome"='public'
- OR "aerodrome_type"='civil'
+ WHEN "aerodrome" = 'public'
+ OR "aerodrome_type" = 'civil'
OR "aerodrome_type" LIKE '%public%'
THEN 'public'
- WHEN "aerodrome"='regional'
- OR "aerodrome_type"='regional'
+ WHEN "aerodrome" = 'regional'
+ OR "aerodrome_type" = 'regional'
THEN 'regional'
- WHEN "aerodrome"='military'
+ WHEN "aerodrome" = 'military'
OR "aerodrome_type" LIKE '%military%'
- OR "military"='airfield'
+ OR "military" = 'airfield'
THEN 'military'
- WHEN "aerodrome"='private'
- OR "aerodrome_type"='private'
+ WHEN "aerodrome" = 'private'
+ OR "aerodrome_type" = 'private'
THEN 'private'
ELSE 'other'
END AS class,
```
Yuri Astrakhan