planetiler/planetiler-core/src/main/java/com/onthegomap/planetiler/archive/TileArchiveMetadata.java

208 wiersze
8.2 KiB
Java
Czysty Zwykły widok Historia

package com.onthegomap.planetiler.archive;
2021-09-18 01:12:24 +00:00
2023-03-18 18:38:04 +00:00
import com.fasterxml.jackson.annotation.JsonAnyGetter;
import com.fasterxml.jackson.annotation.JsonAnySetter;
2023-03-18 18:38:04 +00:00
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.json.JsonMapper;
import com.fasterxml.jackson.databind.ser.std.ToStringSerializer;
import com.onthegomap.planetiler.Profile;
2023-03-18 18:38:04 +00:00
import com.onthegomap.planetiler.config.PlanetilerConfig;
import com.onthegomap.planetiler.geo.GeoUtils;
import com.onthegomap.planetiler.util.BuildInfo;
2023-09-22 01:44:09 +00:00
import com.onthegomap.planetiler.util.LayerAttrStats;
import java.util.LinkedHashMap;
2023-03-18 18:38:04 +00:00
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
import org.locationtech.jts.geom.Coordinate;
2023-03-18 18:38:04 +00:00
import org.locationtech.jts.geom.Envelope;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
2021-09-18 01:12:24 +00:00
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
/**
* Metadata associated with a tile archive.
* <p>
* The default (de-)serialization corresponds to the
* <a href="https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md#metadata">mbtiles spec</a>. As such each
* value is a string.
*/
2023-01-17 12:05:45 +00:00
public record TileArchiveMetadata(
2023-03-18 18:38:04 +00:00
@JsonProperty(NAME_KEY) String name,
@JsonProperty(DESCRIPTION_KEY) String description,
@JsonProperty(ATTRIBUTION_KEY) String attribution,
@JsonProperty(VERSION_KEY) String version,
@JsonProperty(TYPE_KEY) String type,
@JsonProperty(FORMAT_KEY) String format,
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
@JsonSerialize(using = TileArchiveMetadataDeSer.EnvelopeSerializer.class)
@JsonDeserialize(using = TileArchiveMetadataDeSer.EnvelopeDeserializer.class) Envelope bounds,
@JsonSerialize(using = TileArchiveMetadataDeSer.CoordinateSerializer.class)
@JsonDeserialize(using = TileArchiveMetadataDeSer.CoordinateDeserializer.class) Coordinate center,
@JsonProperty(MINZOOM_KEY)
@JsonSerialize(using = ToStringSerializer.class) Integer minzoom,
@JsonProperty(MAXZOOM_KEY)
@JsonSerialize(using = ToStringSerializer.class) Integer maxzoom,
@JsonProperty(JSON_KEY)
@JsonSerialize(using = TileArchiveMetadataDeSer.MetadataJsonSerializer.class)
@JsonDeserialize(using = TileArchiveMetadataDeSer.MetadataJsonDeserializer.class) TileArchiveMetadataJson json,
@JsonAnyGetter
@JsonDeserialize(using = TileArchiveMetadataDeSer.EmptyMapIfNullDeserializer.class) Map<String, String> others,
@JsonProperty(COMPRESSION_KEY) TileCompression tileCompression
2021-09-18 01:12:24 +00:00
) {
2023-03-18 18:38:04 +00:00
public static final String NAME_KEY = "name";
public static final String DESCRIPTION_KEY = "description";
public static final String ATTRIBUTION_KEY = "attribution";
public static final String VERSION_KEY = "version";
public static final String TYPE_KEY = "type";
public static final String FORMAT_KEY = "format";
public static final String BOUNDS_KEY = "bounds";
public static final String CENTER_KEY = "center";
public static final String ZOOM_KEY = "zoom";
public static final String MINZOOM_KEY = "minzoom";
public static final String MAXZOOM_KEY = "maxzoom";
public static final String VECTOR_LAYERS_KEY = "vector_layers";
public static final String COMPRESSION_KEY = "compression";
2023-03-18 18:38:04 +00:00
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
public static final String JSON_KEY = "json";
2023-03-18 18:38:04 +00:00
public static final String MVT_FORMAT = "pbf";
private static final Logger LOGGER = LoggerFactory.getLogger(TileArchiveMetadata.class);
public TileArchiveMetadata(Profile profile, PlanetilerConfig config) {
this(profile, config, null);
2021-09-18 01:12:24 +00:00
}
2023-09-22 01:44:09 +00:00
public TileArchiveMetadata(Profile profile, PlanetilerConfig config, List<LayerAttrStats.VectorLayer> vectorLayers) {
2021-09-18 01:12:24 +00:00
this(
2023-03-18 18:38:04 +00:00
getString(config, NAME_KEY, profile.name()),
getString(config, DESCRIPTION_KEY, profile.description()),
getString(config, ATTRIBUTION_KEY, profile.attribution()),
getString(config, VERSION_KEY, profile.version()),
getString(config, TYPE_KEY, profile.isOverlay() ? "overlay" : "baselayer"),
getString(config, FORMAT_KEY, MVT_FORMAT),
config.bounds().latLon(),
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
new Coordinate(
config.bounds().latLon().centre().getX(),
config.bounds().latLon().centre().getY(),
GeoUtils.getZoomFromLonLatBounds(config.bounds().latLon())
),
2023-03-18 18:38:04 +00:00
config.minzoom(),
config.maxzoom(),
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
vectorLayers == null ? null : new TileArchiveMetadataJson(vectorLayers),
mergeMaps(mapWithBuildInfo(),profile.extraArchiveMetadata()),
config.tileCompression()
2021-09-18 01:12:24 +00:00
);
}
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
// just used for the "internal map"-serialization - ignored by default
@JsonIgnore
@JsonProperty(ZOOM_KEY)
public Double zoom() {
if (center == null) {
return null;
}
final double z = center.getZ();
return Double.isNaN(z) ? null : z;
}
// just used for the "internal map"-serialization - ignored by default
@JsonIgnore
@JsonProperty(VECTOR_LAYERS_KEY)
public List<LayerAttrStats.VectorLayer> vectorLayers() {
return json == null ? null : json.vectorLayers;
}
2023-03-18 18:38:04 +00:00
private static String getString(PlanetilerConfig config, String key, String fallback) {
return config.arguments()
.getString("archive_" + key + "|mbtiles_" + key, "'" + key + "' attribute for tileset metadata", fallback);
}
private static Map<String, String> mapWithBuildInfo() {
Map<String, String> result = new LinkedHashMap<>();
var buildInfo = BuildInfo.get();
if (buildInfo != null) {
if (buildInfo.version() != null) {
result.put("planetiler:version", buildInfo.version());
}
if (buildInfo.githash() != null) {
result.put("planetiler:githash", buildInfo.githash());
}
if (buildInfo.buildTimeString() != null) {
result.put("planetiler:buildtime", buildInfo.buildTimeString());
}
}
return result;
}
2023-03-18 18:38:04 +00:00
/** Sets an extra metadata entry in {@link #others}. */
public TileArchiveMetadata setExtraMetadata(String key, Object value) {
if (key != null && value != null) {
2023-03-18 18:38:04 +00:00
others.put(key, value.toString());
}
return this;
}
2023-03-18 18:38:04 +00:00
/**
* Returns a map with all key-value pairs from this metadata entry, including {@link #others} hoisted to top-level
* keys.
*/
public Map<String, String> toMap() {
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
final JsonMapper mapper = TileArchiveMetadataDeSer.internalMapMapper();
return new LinkedHashMap<>(mapper.convertValue(this, new TypeReference<>() {}));
2023-03-18 18:38:04 +00:00
}
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
/** Returns a copy of this instance with {@link #json} set to {@code layerStats}. */
2023-09-22 01:44:09 +00:00
public TileArchiveMetadata withLayerStats(List<LayerAttrStats.VectorLayer> layerStats) {
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
return withJson(json == null ? TileArchiveMetadataJson.create(layerStats) : json.withLayers(layerStats));
}
/**
* Returns a copy of this instance with {@link #json}'s {@link TileArchiveMetadataJson#vectorLayers()} set to
* {@code layerStats}.
*/
public TileArchiveMetadata withJson(TileArchiveMetadataJson json) {
return new TileArchiveMetadata(name, description, attribution, version, type, format, bounds, center, minzoom,
maxzoom, json, others, tileCompression);
}
public TileArchiveMetadata withTileCompression(TileCompression tileCompression) {
return new TileArchiveMetadata(name, description, attribution, version, type, format, bounds, center, minzoom,
maxzoom, json, others, tileCompression);
}
/*
* few workarounds to make collect unknown fields to others work,
* because @JsonAnySetter does not yet work on constructor/creator arguments
* https://github.com/FasterXML/jackson-databind/issues/3439
*/
private static Map<String,String> mergeMaps(Map<String,String> m1, Map<String,String> m2) {
var result = new TreeMap<>(m1);
result.putAll(m2);
return result;
}
@JsonAnySetter
private void putUnknownFieldsToOthers(String name, String value) {
others.put(name, value);
}
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
public record TileArchiveMetadataJson(
@JsonProperty(VECTOR_LAYERS_KEY) List<LayerAttrStats.VectorLayer> vectorLayers
) {
public TileArchiveMetadataJson withLayers(List<LayerAttrStats.VectorLayer> vectorLayers) {
return TileArchiveMetadataJson.create(vectorLayers);
}
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
public static TileArchiveMetadataJson create(List<LayerAttrStats.VectorLayer> vectorLayers) {
return vectorLayers == null ? null : new TileArchiveMetadataJson(vectorLayers);
}
}
2021-09-18 01:12:24 +00:00
}