planetiler/PLANET.md

Generating a Map of the World

To generate a map of the world using the built-in basemap profile based on OpenMapTiles, you will need a machine with java 16 or later installed and at least 10x as much disk space and 1.5x as much RAM as the planet.osm.pbf file you start from. All testing has been done using Digital Ocean droplets with dedicated vCPUs (referral link) and OpenJDK installed through apt. Flatmap splits work among cores so the more you have, the less time it takes.

1) Choose the Data Source

First decide where to get the planet.osm.pbf file:

• One of the official mirrors
• The AWS Registry of Open Data public S3 mirror (default)
• Or a Daylight Distribution snapshot from Facebook that lags a bit, but includes extra quality/consistency checks, and add-ons like ML-detected roads and buildings. NOTE: you need at least admin.osc.bz2 then combine and re-number using osmium-tool:

  osmium apply-changes daylight.osm.pbf admin.osc.bz2 <buildings.osc.bz2, ...> -o everything.osm.pbf
  osmium renumber everything.osm.pbf -o planet.osm.pbf
  

  This takes about 2.5 hours and needs as much RAM as the planet.osm.pbf size.

2) Run Flatmap

Download the latest release of flatmap.jar.

Then run java -Xms100g -Xmx100g -jar flatmap.jar (replacing 100g with 1.5x the planet.osm.pbf size) with these options:

• --bounds=world to set bounding box to the entire planet
• --nodemap-type=sparsearray to store node locations in a sparse array instead of a sorted table (sorted table is only more efficient for extracts)
• --nodemap-storage=ram to store all node locations in RAM instead of a memory-mapped file (when using ram give the JVM 1.5x the input file size instead of 0.5x when using mmap)
• --download to fetch other data sources automatically
• One of these to point flatmap at your data source:
  • --osm-path=path/to/planet.osm.pbf to point Flatmap at a file you downloaded
  • --osm-url=http://url/of/planet.osm.pbf to download automatically
  • --area=planet to use the file in ./data/sources/planet.osm.pbf or download the latest snapshot from AWS S3 mirror if missing.

Run with --help to see all available arguments.

Example

To generate the tiles shown on https://onthegomap.github.io/flatmap-demo/ I used the planet-211011.osm.pbf (64.7GB) S3 snapshot, then ran Flatmap on a Digital Ocean Memory-Optimized droplet with 16 CPUs, 128GB RAM, and 1.17TB disk running Ubuntu 21.04 x64 in the nyc3 location.

First, I installed java 17 jre and screen:

apt-get update && apt-get install -y openjdk-17-jre-headless screen

Then I added a script runworld.sh to run with 100GB of RAM:

#!/usr/bin/env bash
set -e
java -Xmx100g -Xms100g \
  -XX:OnOutOfMemoryError="kill -9 %p" \
  -jar flatmap.jar \
  `# Download the latest planet.osm.pbf from s3://osm-pds bucket` \
  --area=planet --bounds=world --download \
  `# Accelerate the download by fetching the 10 1GB chunks at a time in parallel` \
  --download-threads=10 --download-chunk-size-mb=1000 \
  `# Also download name translations from wikidata` \
  --fetch-wikidata \
  `# Personal preference overrides to default OpenMapTiles schema` \
  --transportation-name-brunnel=false --transportation-z13-paths=true \
  --mbtiles=output.mbtiles \
  --nodemap-type=sparsearray --nodemap-storage=ram 2>&1 | tee logs.txt

Then I ran this in the background using screen, so it would continue if my shell exited:

screen -d -m "./runworld.sh"
tail -f logs.txt

It took 3h21m (including 12 minutes downloading source data) to generate a 99GB output.mbtiles file. See the full logs from this run or this summary that it printed at the end. Notice that it spent almost an hour emitting z13 tiles. That is because the default basemap profile merges nearby building polygons at z13 which is very expensive. You can disable this behavior by setting --building-merge-z13=false.

3:21:03 DEB [mbtiles] - Tile stats:
3:21:03 DEB [mbtiles] - z0 avg:71k max:71k
3:21:03 DEB [mbtiles] - z1 avg:171k max:192k
3:21:03 DEB [mbtiles] - z2 avg:258k max:449k
3:21:03 DEB [mbtiles] - z3 avg:117k max:479k
3:21:03 DEB [mbtiles] - z4 avg:51k max:541k
3:21:03 DEB [mbtiles] - z5 avg:23k max:537k
3:21:03 DEB [mbtiles] - z6 avg:14k max:354k
3:21:03 DEB [mbtiles] - z7 avg:11k max:451k
3:21:03 DEB [mbtiles] - z8 avg:6.5k max:356k
3:21:03 DEB [mbtiles] - z9 avg:6k max:485k
3:21:03 DEB [mbtiles] - z10 avg:2.7k max:285k
3:21:03 DEB [mbtiles] - z11 avg:1.3k max:168k
3:21:03 DEB [mbtiles] - z12 avg:741 max:247k
3:21:03 DEB [mbtiles] - z13 avg:388 max:286k
3:21:03 DEB [mbtiles] - z14 avg:340 max:1.7M
3:21:03 DEB [mbtiles] - all avg:395 max:0
3:21:03 DEB [mbtiles] -  # features: 2,832,396,934
3:21:03 DEB [mbtiles] -     # tiles: 264,204,266
3:21:03 INF [mbtiles] - Finished in 4,668s cpu:66,977s avg:14.3

3:21:03 INF - Finished in 12,064s cpu:156,169s avg:12.9

3:21:03 INF - FINISHED!
3:21:03 INF - ----------------------------------------
3:21:03 INF - 	overall	12,064s cpu:156,169s avg:12.9
3:21:03 INF - 	download	169s cpu:1,070s avg:6.3
3:21:03 INF - 	wikidata	553s cpu:3,825s avg:6.9
3:21:03 INF - 	lake_centerlines	0.9s cpu:2s avg:1.8
3:21:03 INF - 	water_polygons	96s cpu:1,150s avg:12
3:21:03 INF - 	natural_earth	6s cpu:21s avg:3.7
3:21:03 INF - 	osm_pass1	921s cpu:5,177s avg:5.6
3:21:03 INF - 	osm_pass2	5,234s cpu:73,527s avg:14
3:21:03 INF - 	boundaries	14s cpu:18s avg:1.3
3:21:03 INF - 	sort	407s cpu:4,403s avg:10.8
3:21:03 INF - 	mbtiles	4,668s cpu:66,977s avg:14.3
3:21:03 INF - ----------------------------------------
3:21:03 INF - 	features	192GB
3:21:03 INF - 	mbtiles	99GB

Then to generate the extract for the demo I ran:

# install node and tilelive-copy
curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
apt-get install -y nodejs
npm install -g @mapbox/tilelive @mapbox/mbtiles
# Extract z0-4 for the world
tilelive-copy --minzoom=0 --maxzoom=4 --bounds=-180,-90,180,90 output.mbtiles demo.mbtiles
# Extract z0-14 for just southern New England
tilelive-copy --minzoom=0 --maxzoom=14 --bounds=-73.6346,41.1055,-69.5464,42.9439 output.mbtiles demo.mbtiles