Wykres commitów

11 Commity (2aaa288779867d85d54588766fb3e0c79e0cfbaf)

Autor SHA1 Wiadomość Data
Mike Barry 2aaa288779 im4gn 2024-01-20 08:58:37 -05:00
Mike Barry 2b918abd76 16gb benchmark 2024-01-19 06:49:41 -05:00
Michael Barry 14b217d6f6
Fix feature merge consistent ordering (#789) 2024-01-14 12:08:20 -05:00
Michael Barry 96eae6110b
Archive compare (#780) 2024-01-10 05:21:03 -05:00
Björn Bilger c480b35f1c
Add support for "files"-archive (#761)
* Add support for "files"-archive

i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf

in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"

Fixes #536

* default to files format

...if no explict format query param given, path ends with a slash, or no extension given

* output metadata.json in files-archive

and refactor TileArchiveMetadata
1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format
2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner
3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format

* add support for custom tile scheme in files-archive

{z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.:
- different order: {x}/{y}/{z}.pbf
- with intermediate dirs: {x}/a/{y}/b/{z}.pbf
- with different extension: {z}/{y}/{y}.pbf.gz

instead of {x} and {y}, {xs} and {xy} can be used which breaks up
x and y into 2 directories each and ensures that each directory has <1000 children

* fix issues with multiple writers

1. call finish archive only once after all writers are finished
   ...and not every time a writer finishes
2. log "zoom-progress" for the first tile write only
   (Finished z11 ... now starting z12)
3. remove file/dir-size progress logger bottleneck for files archive
   => each archive now reports the bytes written, which also fixes
   the issues of stream-archives reporting the size incorrectly
4. introduce printStats-hook on archive-level

* add async file write support to files archive

...allow to use virtual threads ExecturService (bound only!) for tile writing

also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual

* Revert "add async file write support to files archive"

This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0.

* few improvements

- extract TileSchemeEncoding
- use Counter.MultithreadCounter rather than LongAdder to count bytes written
- add some JavaDoc

* simplify files archive usage

1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf
2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI

* few more adjustments according to PR feeback

1. use WriteableTileArchive#bytesWritten in summmary as well
2. call WriteableTileArchive#init in a safer manner

..and a few more adjustments

* more PR feedback
2024-01-02 20:37:49 -05:00
Michael Barry cbb092a1f7
Compute layer attr stats from actual vector tile features (#752) 2023-12-14 19:26:27 -05:00
Michael Barry 2f86ea12ae
Utilities to reduce tile size (#669) 2023-09-24 08:10:47 -04:00
Michael Barry 1f23b55274
Tile stats (#656) 2023-09-21 21:44:09 -04:00
Björn Bilger db796e1720
Add support for simple file output streams: CSV, JSON, protobuf (#639) 2023-08-23 20:24:27 -04:00
Michael Barry 74db638dbc
Expose pmtiles writer (#520) 2023-03-18 14:38:04 -04:00
Michael Barry 9a704e773e
Convert get all tile coords to iterator (#463) 2023-02-05 14:16:05 -05:00