* `make start-postserve PPORT=8000`
* `make start-tileserver TPORT=9000`
Note that both ports need to be the same on host and inside the docker container because the services include a URL to the tiles as part of their manifest.
Add `getmvt` sql function as part of the build-sql make step. The function will be imported into PG during the `import-sql` step after all other SQL is done. This is a noop for the `generate-tiles` approach.
* make sure all required dirs are initialized before running any of the docker commands. Otherwise docker will create them as root.
* minor docs cleanup
Generate multiple SQL files to be imported in parallel.
The files will respect the cross-layer dependencies,
so they can be all ingested at the same time.
* Adds a script to downloads multiple areas and compute their test parameters
* added a large test that uses a combined 76MB file with equatorial-guinea, liechtenstein, district-of-columbia, greater-london
* cache wikidata downloads
Update to tools v5. See https://github.com/openmaptiles/openmaptiles-tools/releases/tag/v5.0.0 for the list of all changes. Other OMT-repo specific changes:
* removes `import-osm` docker usage, replacing it with `openmaptiles-tools`
* quickstart builds faster because it uses postgres with preloaded water, natural earth, and lake centerlines tables.
### Makefile targets
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* `db-start-preloaded` is the same as `db-start`, except that it uses `postgis-preloaded` -- an image with preloaded water, natural-earth, and lake centerline data
* `db-start` will not recreate the container if it already exists -- this way if it was started as preloaded, it will not be rebuilt.
* better output messages
### Quickstart
* uses `postgis-preloaded` image by default to make quickstart quicker. To start with a clean db, pass 2 parameters to quickstart, e.g. `./quickstart.sh albania empty`
* allow postgres image to be overwritten with an env var
* allow DIFF_MODE var to be overwritten with an env var
* add /mapping and /cache dirs into tools image
* make `build-sql` target explicit rather than relying on a filename
* `tools-dev` will open a shell in a docker to experiment and debug (instead of `import-sql-dev` and `import-osm-dev`)
* separated `start-maputnik` from `start-postserve`
* renamed `clean-docker` into `db-destroy` to make it more explicit
* cleaner `db-start`, `db-stop`, `db-destroy` targets
* better output messages
This is a partial migration of https://github.com/openmaptiles/openmaptiles/pull/785
* Use `import-data` instead of `import-lakelines`, `import-water`, and `import-natural-earth`
* Upgrade docker-compose.yml to version 2.3 (allows some extra env var usage in yaml file itself)
* Remove `openmaptiles-tools:latest` usage -- no longer needed, can use current version 4.1
* `db-start` does not do a container recreation in case docker-compose.yml definition has changed.
* a few minor cleanups in quickstart.sh
- Add `download-osmfr` and `download-bbbike` targets
- Port `DC_OPTS` to Windows
- Use make conditions instead of shell. Also simplifies `make -n` output
- Use remote Docker machine's IP, if defined, rather than `localhost` in `http://...` messages. Also applicable for Docker Toolbox for Windows.
- Align texts in `make help` output
- Update and bug-fix in `make remove-docker-images`
The implementation adds the `DC_PROJECT` parameter. It can also be set by an environment variable in the hosting shell. The environment variable can be overwritten by a make parameter, including `DC_PROJECT=` which restores the automatic project name.
#### _NOTE:_
It may be worthwhile to review the following make targets that are currently not impacted by the PR:
- `make start-postserve`
- `make list-docker-images`
- `make remove-docker-images`
- `make docker-unnecessary-clean`
This PR was suggested by @nyurik:
> zstadler 3:30 PM
Is it possible to run two instances of openmaptiles, and the postgis container in particular, on the same Linux host?
nyurik 4:17 PM
use docker-compose --project-name -- that should allow you to run everything in parallel
4:17
might need to update make file
4:17
btw, that would be a good PR for makefile -- to specify --project-name based on the current DIR name, but so that it can be overwritten by a makefile param
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
import-wikidata should run before import-sql, and v4.1 allows for that.
Also it optimizes the wd_names table to just be a simple Wikidata ID -> labels lookup, with a proper index.
Minor other changes:
* `test-perf-null` target is now part of the Makefile
* `./data/osmstat.txt` is no longer created
* area download file now in this format: `${osm_area}-latest.osm.pbf`
* Switch OMT to use the new tools v4.0.0
* borders are dynamically generated from the PBF file instead of downloading a prepared CSV file
* all tools are executed as current user instead of root, thus files are easier to modify/delete if needed
* all data is stored in the local file system instead of docker volumes (Docker currently has a limitation of non-root operation for internal volumes). This also makes it easier to examine and test it.
* New `init-dirs` make target creates all the needed dirs - `build, data, cache`
* `make clean` deletes the whole `build` dir instead of individual files.
* `clean-docker` for backward compatibility deletes `cache` dirs (it used to be a volume)
* all `psql` calls are now done with `ON_ERROR_STOP=1`
* got rid of `pgclimb-*` targets -- same results can be done with `psql` (`pgclimb-list-views` & `pgclimb-list-tables` renamed to `list-views` and `list-tables`)
Make a few more mappings declarative, and removes values declared in both SQL and the yaml file.
Here's the diff comparing `build/tileset.sql` in master vs the new PR. The changes are mostly stylistic, except when a nested `if` statement is expanded into individual `if ... and ...` conditions (logically identical)
```diff
55c55
diff --git a/build/tileset.sql b/build/tileset.sql
index 4e59357..7c6c444 100644
--- a/build/tileset.sql
+++ b/build/tileset.sql
@@ -52,7 +52,7 @@ CREATE INDEX IF NOT EXISTS osm_ocean_polygon_gen4_idx ON osm_ocean_polygon_gen4
CREATE OR REPLACE FUNCTION water_class(waterway TEXT) RETURNS TEXT AS $$
SELECT CASE
WHEN "waterway" IN ('', 'lake') THEN 'lake'
- WHEN "waterway"='dock' THEN 'dock'
+ WHEN "waterway" = 'dock' THEN 'dock'
ELSE 'river'
END;
$$ LANGUAGE SQL IMMUTABLE;
@@ -1813,24 +1813,41 @@ CREATE OR REPLACE FUNCTION highway_class(highway TEXT, public_transport TEXT, co
WHEN "highway" IN ('tertiary', 'tertiary_link') THEN 'tertiary'
WHEN "highway" IN ('unclassified', 'residential', 'living_street', 'road') THEN 'minor'
WHEN "highway" IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor')
- OR "public_transport"='platform'
+ OR "public_transport" = 'platform'
THEN 'path'
- WHEN "highway"='service' THEN 'service'
- WHEN "highway"='track' THEN 'track'
- WHEN "highway"='raceway' THEN 'raceway'
- WHEN highway = 'construction' THEN CASE
- WHEN construction IN ('motorway', 'motorway_link') THEN 'motorway_construction'
- WHEN construction IN ('trunk', 'trunk_link') THEN 'trunk_construction'
- WHEN construction IN ('primary', 'primary_link') THEN 'primary_construction'
- WHEN construction IN ('secondary', 'secondary_link') THEN 'secondary_construction'
- WHEN construction IN ('tertiary', 'tertiary_link') THEN 'tertiary_construction'
- WHEN construction IN ('', 'unclassified', 'residential', 'living_street', 'road') THEN 'minor_construction'
- WHEN construction IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor')
- OR public_transport = 'platform' THEN 'path_construction'
- WHEN construction = 'service' THEN 'service_construction'
- WHEN construction = 'track' THEN 'track_construction'
- WHEN construction = 'raceway' THEN 'raceway_construction'
- END
+ WHEN "highway" = 'service' THEN 'service'
+ WHEN "highway" = 'track' THEN 'track'
+ WHEN "highway" = 'raceway' THEN 'raceway'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('motorway', 'motorway_link')
+ THEN 'motorway_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('trunk', 'trunk_link')
+ THEN 'trunk_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('primary', 'primary_link')
+ THEN 'primary_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('secondary', 'secondary_link')
+ THEN 'secondary_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('tertiary', 'tertiary_link')
+ THEN 'tertiary_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" IN ('', 'unclassified', 'residential', 'living_street', 'road')
+ THEN 'minor_construction'
+ WHEN "highway" = 'construction'
+ AND ("construction" IN ('pedestrian', 'path', 'footway', 'cycleway', 'steps', 'bridleway', 'corridor') OR "public_transport" = 'platform')
+ THEN 'path_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" = 'service'
+ THEN 'service_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" = 'track'
+ THEN 'track_construction'
+ WHEN "highway" = 'construction'
+ AND "construction" = 'raceway'
+ THEN 'raceway_construction'
END;
$$ LANGUAGE SQL IMMUTABLE;
@@ -4073,6 +4090,12 @@ RETURNS TEXT AS $$
WHEN "subclass" IN ('fast_food', 'food_court') THEN 'fast_food'
WHEN "subclass" IN ('park', 'bbq') THEN 'park'
WHEN "subclass" IN ('bus_stop', 'bus_station') THEN 'bus'
+ WHEN ("subclass" = 'station' AND "mapping_key" = 'railway')
+ OR "subclass" IN ('halt', 'tram_stop', 'subway')
+ THEN 'railway'
+ WHEN "subclass" = 'station'
+ AND "mapping_key" = 'aerialway'
+ THEN 'aerialway'
WHEN "subclass" IN ('subway_entrance', 'train_station_entrance') THEN 'entrance'
WHEN "subclass" IN ('camp_site', 'caravan_site') THEN 'campsite'
WHEN "subclass" IN ('laundry', 'dry_cleaning') THEN 'laundry'
@@ -4082,7 +4105,7 @@ RETURNS TEXT AS $$
WHEN "subclass" IN ('hotel', 'motel', 'bed_and_breakfast', 'guest_house', 'hostel', 'chalet', 'alpine_hut', 'dormitory') THEN 'lodging'
WHEN "subclass" IN ('chocolate', 'confectionery') THEN 'ice_cream'
WHEN "subclass" IN ('post_box', 'post_office') THEN 'post'
- WHEN "subclass"='cafe' THEN 'cafe'
+ WHEN "subclass" = 'cafe' THEN 'cafe'
WHEN "subclass" IN ('school', 'kindergarten') THEN 'school'
WHEN "subclass" IN ('alcohol', 'beverages', 'wine') THEN 'alcohol_shop'
WHEN "subclass" IN ('bar', 'nightclub') THEN 'bar'
@@ -4098,9 +4121,6 @@ RETURNS TEXT AS $$
WHEN "subclass" IN ('bag', 'clothes') THEN 'clothing_store'
WHEN "subclass" IN ('swimming_area', 'swimming') THEN 'swimming'
WHEN "subclass" IN ('castle', 'ruins') THEN 'castle'
- WHEN (subclass = 'station' AND mapping_key = 'railway')
- OR (subclass IN ('halt', 'tram_stop', 'subway')) THEN 'railway'
- WHEN (subclass = 'station' AND mapping_key = 'aerialway') THEN 'aerialway'
ELSE subclass
END;
$$ LANGUAGE SQL IMMUTABLE;
@@ -4301,22 +4321,22 @@ $$
COALESCE(NULLIF(name_de, ''), name, name_en) AS name_de,
tags,
CASE
- WHEN "aerodrome"='international'
- OR "aerodrome_type"='international'
+ WHEN "aerodrome" = 'international'
+ OR "aerodrome_type" = 'international'
THEN 'international'
- WHEN "aerodrome"='public'
- OR "aerodrome_type"='civil'
+ WHEN "aerodrome" = 'public'
+ OR "aerodrome_type" = 'civil'
OR "aerodrome_type" LIKE '%public%'
THEN 'public'
- WHEN "aerodrome"='regional'
- OR "aerodrome_type"='regional'
+ WHEN "aerodrome" = 'regional'
+ OR "aerodrome_type" = 'regional'
THEN 'regional'
- WHEN "aerodrome"='military'
+ WHEN "aerodrome" = 'military'
OR "aerodrome_type" LIKE '%military%'
- OR "military"='airfield'
+ OR "military" = 'airfield'
THEN 'military'
- WHEN "aerodrome"='private'
- OR "aerodrome_type"='private'
+ WHEN "aerodrome" = 'private'
+ OR "aerodrome_type" = 'private'
THEN 'private'
ELSE 'other'
END AS class,
```
quicker and cleaner diagram image generation.
Remove etl-graph and mapping-graph targets - redundant
Also, the obsolete "fields" is still in Imposm's code and both names are accepted,
but "fields" is not documented anywhere (PR submitted), and could be removed at any moment.
Our docs were not supporting it until this PR, so renaming it at the same time.
Several images have been updated due to a more inclusive mapping scan
Requires https://github.com/openmaptiles/openmaptiles-tools/pull/147 (merged)
* Use _resolve_wikidata in layer mapping.yaml
Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.
This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.
The next tools version will remove it when generating imposm mapping file.
* Migrate to new Wikidata importer
Uses latest tools to populate the wd_names table
during the quickstart. This can be merged already,
or we can wait for the next tools version.
* Use unified tools version for all images
* do not start postserve as part of quickstart, but added a help message how to start it
* wait for SQL start with pgwait
Seems like etl and mapping diagrams have been neglected
for a long time. Now it regenerates the files and places
them in the source dir.
This PR also fixes two broken files:
* layers/aerodrome_label/mapping_diagram.png
* layers/housenumber/mapping_diagram.png
They were generated using the newest tools version with the fix
https://github.com/openmaptiles/openmaptiles-tools/pull/65
Yuri Astrakhan