diff --git a/.github/workflows/docs_mlc_config.json b/.github/workflows/docs_mlc_config.json index c5c444af..743dce0e 100644 --- a/.github/workflows/docs_mlc_config.json +++ b/.github/workflows/docs_mlc_config.json @@ -1,7 +1,7 @@ { "ignorePatterns": [ { - "pattern": "^https://github.com/onthegomap/flatmap/.*$" + "pattern": "^https://github.com/onthegomap/flatmap.*$" }, { "pattern": "^https://onthegomap.github.io/.*$" diff --git a/PLANET.md b/PLANET.md index 7587ff8b..63305cfc 100644 --- a/PLANET.md +++ b/PLANET.md @@ -1,10 +1,10 @@ # Generating a Map of the World -To generate a map of the world using the built-in [basemap profile](flatmap-basemap) based -on [OpenMapTiles](https://github.com/openmaptiles/openmaptiles), you will need a machine with java 16 or later installed -and at least 10x as much disk space and 1.5x as much RAM as the `planet.osm.pbf` file you start from. All testing has -been done using Digital Ocean droplets with dedicated vCPUs ([referral link](https://m.do.co/c/a947e99aab25)) and -OpenJDK installed through `apt`. Flatmap splits work among cores so the more you have, the less time it takes. +To generate a map of the world using the built-in [basemap profile](flatmap-basemap), you will need a machine with Java +16 or later installed and at least 10x as much disk space and 1.5x as much RAM as the `planet.osm.pbf` file you start +from. All testing has been done using Digital Ocean droplets with dedicated +vCPUs ([referral link](https://m.do.co/c/a947e99aab25)) and OpenJDK 17 installed through `apt`. Flatmap splits work +among available CPUs so the more you have, the less time it takes. ### 1) Choose the Data Source @@ -12,14 +12,15 @@ First decide where to get the `planet.osm.pbf` file: - One of the [official mirrors](https://wiki.openstreetmap.org/wiki/Planet.osm) - The [AWS Registry of Open Data](https://registry.opendata.aws/osm/) public S3 mirror (default) -- Or a [Daylight Distribution](https://daylightmap.org/) snapshot from Facebook that lags a bit, but includes extra - quality/consistency checks, and add-ons like ML-detected roads and buildings. NOTE: you need at least - `admin.osc.bz2` then combine and re-number using [osmium-tool](https://osmcode.org/osmium-tool/): +- Or a [Daylight Distribution](https://daylightmap.org/) snapshot from Facebook that includes extra quality/consistency + checks, and add-ons like ML-detected roads and buildings. Combine add-ons and re-number + using [osmium-tool](https://osmcode.org/osmium-tool/): ```bash osmium apply-changes daylight.osm.pbf admin.osc.bz2 -o everything.osm.pbf osmium renumber everything.osm.pbf -o planet.osm.pbf ``` - This takes about 2.5 hours and needs as much RAM as the `planet.osm.pbf` size. + NOTE: you need at least `admin.osc.bz2` for the `boundary` layer to show. This takes about 2.5 hours and needs as much + RAM as the `planet.osm.pbf` size. ### 2) Run Flatmap @@ -29,19 +30,26 @@ Then run `java -Xms100g -Xmx100g -jar flatmap.jar` (replacing `100g` with 1.5x t with these options: - `--bounds=world` to set bounding box to the entire planet -- `--nodemap-type=sparsearray` to store node locations in a sparse array instead of a sorted table (sorted table is only - more efficient for extracts) -- `--nodemap-storage=ram` to store all node locations in RAM instead of a memory-mapped file (when using `ram` give the - JVM 1.5x the input file size instead of 0.5x when using `mmap`) +- `--nodemap-type=sparsearray` to store node locations in a sparse array instead of a sorted table - `sortedtable` is + more efficient when there are large gaps in ID spaces (i.e. extracts) and `sparsearray` is more efficient with no/few + ID gaps (planet, or renumbered extracts). +- `--nodemap-storage=ram` to store all node locations in RAM instead of a memory-mapped file - when using `ram` give the + JVM 1.5x the input file size instead of 0.5x when using `mmap` - `--download` to fetch [other data sources](NOTICE.md#data) automatically - One of these to point flatmap at your data source: - `--osm-path=path/to/planet.osm.pbf` to point Flatmap at a file you downloaded - `--osm-url=http://url/of/planet.osm.pbf` to download automatically + - `--osm-url=s3:211011` to download a specific snapshot from the AWS Registry of Open Data or `--osm-url=s3:latest` to + download the latest snapshot - `--area=planet` to use the file in `./data/sources/planet.osm.pbf` or download the latest snapshot from AWS S3 mirror if missing. Run with `--help` to see all available arguments. +NOTE: The default basemap profile merges nearby buildings at zoom-level 13 (for example, +see [Boston](https://onthegomap.github.io/flatmap-demo/#13.08/42.35474/-71.06597)). This adds about 50% to the planet +generation time and can be disabled using `--building-merge-z13=false`. + ## Example To generate the tiles shown on https://onthegomap.github.io/flatmap-demo/ I used the `planet-211011.osm.pbf` (64.7GB) S3 @@ -68,8 +76,6 @@ java -Xmx100g -Xms100g \ --download-threads=10 --download-chunk-size-mb=1000 \ `# Also download name translations from wikidata` \ --fetch-wikidata \ - `# Personal preference overrides from OpenMapTiles schema (these are default now)` \ - --transportation-name-brunnel=false --transportation-z13-paths=true \ --mbtiles=output.mbtiles \ --nodemap-type=sparsearray --nodemap-storage=ram 2>&1 | tee logs.txt ``` @@ -82,9 +88,7 @@ tail -f logs.txt ``` It took 3h21m (including 12 minutes downloading source data) to generate a 99GB `output.mbtiles` file. See -the [full logs](planet-logs/v0.1.0-planet-do-16cpu-128gb.txt) from this run or this summary that it printed at the end. -Notice that it spent almost an hour emitting z13 tiles. That is because the default basemap profile merges nearby -building polygons at z13 which is very expensive. You can disable this behavior by setting `--building-merge-z13=false`. +the [full logs](planet-logs/v0.1.0-planet-do-16cpu-128gb.txt) from this run or this summary that it printed at the end: ``` 3:21:03 DEB [mbtiles] - Tile stats: @@ -128,7 +132,7 @@ building polygons at z13 which is very expensive. You can disable this behavior 3:21:03 INF - mbtiles 99GB ``` -Then to generate the extract for [the demo](https://onthegomap.github.io/flatmap-demo/) I ran: +To generate the extract for [the demo](https://onthegomap.github.io/flatmap-demo/) I ran: ```bash # install node and tilelive-copy @@ -140,3 +144,6 @@ tilelive-copy --minzoom=0 --maxzoom=4 --bounds=-180,-90,180,90 output.mbtiles de # Extract z0-14 for just southern New England tilelive-copy --minzoom=0 --maxzoom=14 --bounds=-73.6346,41.1055,-69.5464,42.9439 output.mbtiles demo.mbtiles ``` + +Then I ran [extract.sh](https://github.com/onthegomap/flatmap-demo/blob/main/extract.sh) in the flatmap-demo repo to +extract tiles from the mbtiles file to disk. diff --git a/README.md b/README.md index 72d80306..c7531190 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,9 @@ download regularly-updated tilesets. - [ ] Poi layer `agg_stop` tag - [ ] Get `flatmap-core` into Maven Central - [ ] Remove geotools dependency for reading shapefiles (not in Maven Central) -- [ ] Remove graphhopper dependency for reading OSM files +- [ ] Remove graphhopper dependency for reading OSM files, and + use [LocationsOnWays](https://blog.jochentopf.com/2016-04-20-node-locations-on-ways.html) to skip node location + storage when present and reduce memory requirement by 70%. - [ ] "Sparse mode" to only store node and relation data for elements used by a profile - [ ] Support zoom levels higher than 14 - [ ] Handle nodes and relations in relations (only ways handled now) diff --git a/flatmap-examples/README.md b/flatmap-examples/README.md index 7e417fca..3f2403e0 100644 --- a/flatmap-examples/README.md +++ b/flatmap-examples/README.md @@ -23,7 +23,8 @@ First, make a copy of this example project. It contains: - `com.onthegomap:flatmap-core` test dependency for test utilities - `maven-assembly-plugin` build plugin configuration to create a single executable jar file from `mvn package` goal command -- `child.pom.xml` exists for the parent pom.xml to treat this as a child project, you can remove it. +- `child.pom.xml` exists for the parent pom.xml to treat this as a child project, you can remove it to run as a + standalone project - [src/main/java/com/onthegomap/flatmap/examples](src/main/java/com/onthegomap/flatmap/examples) - some minimal example map profiles: - [ToiletsOverlay](src/main/java/com/onthegomap/flatmap/examples/ToiletsOverlay.java) - demonstrates how to build a @@ -77,7 +78,7 @@ at zoom level 12 and above: @Override public void processFeature(SourceFeature sourceFeature, FeatureCollector features) { if (sourceFeature.isPoint() && sourceFeature.hasTag("amenity", "toilets")) { - features.point("toilets") // layer name="toilets" + features.point("toilets") // create a point in layer named "toilets" .setMinZoom(12) .setAttr("customers_only", sourceFeature.hasTag("access", "customers")) .setAttr("indoor", sourceFeature.getBoolean("indoor")) @@ -87,7 +88,7 @@ public void processFeature(SourceFeature sourceFeature, FeatureCollector feature } ``` -Next, add a `main` entrypoint method in the same or another class that +Next, add a `main` entrypoint that uses [FlatmapRunner](../flatmap-core/src/main/java/com/onthegomap/flatmap/FlatmapRunner.java) to define input sources and default input/output paths: @@ -141,6 +142,8 @@ public void unitTest() { ); List mapFeatures = TestUtils.processSourceFeature(node, profile); // Then inspect attributes of each of vector tile fetures emitted... + assertEquals(1, mapFeatures.length); + assertEquals(12, mapFeatures.get(0).getMinZoom()); } ``` @@ -173,8 +176,8 @@ for a complete unit and integration test. Check out: - The other [minimal examples](./src/main/java/com/onthegomap/flatmap/examples) -- The [basemap profile](../flatmap-basemap) for a full-featured example of a complex profile with processing broken-out - into a handler per-layer +- The [basemap profile](../flatmap-basemap) for a full-featured example of a complex profile with processing broken out + into a handler per layer - [FlatmapRunner](../flatmap-core/src/main/java/com/onthegomap/flatmap/FlatmapRunner.java) for more options when invoking the program - [FeatureCollector](../flatmap-core/src/main/java/com/onthegomap/flatmap/FeatureCollector.java)