Migrate to new Wikidata importer (#735)

* Use _resolve_wikidata in layer mapping.yaml

Mark all tables that should not be populated with the Wikidata
international labels with a special OMT-specific flag.

This should be ok to merge even before the new tools version
is used because imposm seems to ignore anything it doesn't understand.

The next tools version will remove it when generating imposm mapping file.

* Migrate to new Wikidata importer

Uses latest tools to populate the wd_names table
during the quickstart.  This can be merged already,
or we can wait for the next tools version.

.gitignore

@@ -11,9 +11,6 @@ quickstart.log
 # imput / output data
 data/*
 
-# input wikidata
-wikidata/*
-
 # generated source files
 build/*
 

Makefile

@@ -226,9 +226,10 @@ list:
 download-geofabrik-list:
 	docker-compose run $(DC_OPTS) import-osm  ./download-geofabrik-list.sh
 
-.PHONY: download-wikidata
-download-wikidata:
-	mkdir -p wikidata && docker-compose run $(DC_OPTS) --entrypoint /usr/src/app/download-gz.sh import-wikidata
+.PHONY: import-wikidata
+import-wikidata:
+	# FIXME:   switch to openmaptiles-tools after v3.2+ is published
+	docker-compose run $(DC_OPTS) openmaptiles-tools-latest import-wikidata openmaptiles.yaml
 
 .PHONY: psql-list-tables
 psql-list-tables:

README.md

@@ -106,16 +106,6 @@ docker-compose run import-lakelines
 docker-compose run import-osmborder
 ```
 
-**[Optional]**
-Import latest Wikidata. If an OSM feature has [Key:wikidata](https://wiki.openstreetmap.org/wiki/Key:wikidata), OpenMapTiles check corresponding item in Wikidata and use its [labels](https://www.wikidata.org/wiki/Help:Label) for languages listed in [openmaptiles.yaml](openmaptiles.yaml). So the generated vector tiles includes multi-languages in name field.
-
-Beware that current [Wikidata dump](https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz) is more than 55GB, it takes time to download and import it. If you just want to have a quickstart on OpenMapTiles, just skip this step.
-
-```bash
-make download-wikidata
-docker-compose run import-wikidata
-```
-
 [Download OpenStreetMap data extracts](http://download.geofabrik.de/) and store the PBF file in the `./data` directory.
 
 ```bash
@@ -130,6 +120,14 @@ wget http://download.geofabrik.de/europe/albania-latest.osm.pbf
 docker-compose run import-osm
 ```
 
+Import latest Wikidata. If an OSM feature has [Key:wikidata](https://wiki.openstreetmap.org/wiki/Key:wikidata), OpenMapTiles check corresponding item in Wikidata and use its [labels](https://www.wikidata.org/wiki/Help:Label) for languages listed in [openmaptiles.yaml](openmaptiles.yaml). So the generated vector tiles includes multi-languages in name field.
+
+This step uses [Wikidata Query Service](https://query.wikidata.org) to download just the Wikidata IDs that already exist in the database.
+
+```bash
+make import-wikidata
+```
+
 ### Work on Layers
 
 Each time you modify layer SQL code run `make` and `make import-sql`.

docker-compose.yml

@@ -78,16 +78,18 @@ services:
     volumes:
       - .:/tileset
       - ./build:/sql
-  import-wikidata:
-    image: "openmaptiles/import-wikidata:${TOOLS_VERSION}"
+  openmaptiles-tools:
+    image: "openmaptiles/openmaptiles-tools:${TOOLS_VERSION}"
     env_file: .env
-    command: import-wikidata
     networks:
       - postgres_conn
     volumes:
-      - ./wikidata:/import
-  openmaptiles-tools:
-    image: "openmaptiles/openmaptiles-tools:${TOOLS_VERSION}"
+      - .:/tileset
+      - ./build:/sql
+  openmaptiles-tools-latest:
+    # This target exists for experimental tools that have not yet been published.
+    # Do not use this for production.
+    image: "openmaptiles/openmaptiles-tools:latest"
     env_file: .env
     networks:
       - postgres_conn

quickstart.sh

@@ -134,7 +134,6 @@ echo "====> : Making directories - if they don't exist ( ./build ./data ./pgdata
 mkdir -p pgdata
 mkdir -p build
 mkdir -p data
-mkdir -p wikidata
 
 echo " "
 echo "-------------------------------------------------------------------------------------"
@@ -237,14 +236,6 @@ echo "      : The OpenstreetMap data license: https://www.openstreetmap.org/copy
 echo "      : Thank you OpenStreetMap Contributors ! "
 docker-compose run $DC_OPTS import-osm
 
-echo " "
-echo "-------------------------------------------------------------------------------------"
-echo "====> : Start importing Wikidata: ./wikidata/latest-all.json.gz -> PostgreSQL"
-echo "      : Source code: https://github.com/openmaptiles/openmaptiles-tools/tree/master/docker/import-wikidata "
-echo "      : The Wikidata license: https://www.wikidata.org/wiki/Wikidata:Database_download/en#License "
-echo "      : Thank you Wikidata Contributors ! "
-docker-compose run $DC_OPTS import-wikidata
-
 echo " "
 echo "-------------------------------------------------------------------------------------"
 echo "====> : Start SQL postprocessing:  ./build/tileset.sql -> PostgreSQL "
@@ -259,6 +250,13 @@ echo "--------------------------------------------------------------------------
 echo "====> : Analyze PostgreSQL tables"
 make psql-analyze
 
+echo " "
+echo "-------------------------------------------------------------------------------------"
+echo "====> : Start importing Wikidata: Wikidata Query Service -> PostgreSQL"
+echo "      : The Wikidata license: CC0 - https://www.wikidata.org/wiki/Wikidata:Main_Page "
+echo "      : Thank you Wikidata Contributors ! "
+make import-wikidata
+
 echo " "
 echo "-------------------------------------------------------------------------------------"
 echo "====> : Testing PostgreSQL tables to match layer definitions metadata"