Updated docs, remove download/orthophoto.tif endpoint, migration guide

pull/124/head
Piero Toffanin 2020-09-09 14:42:55 -04:00
rodzic ff21268744
commit 89584ee50b
16 zmienionych plików z 70 dodań i 3303 usunięć

Wyświetl plik

@ -5,7 +5,7 @@ EXPOSE 3000
USER root
RUN curl --silent --location https://deb.nodesource.com/setup_10.x | bash -
RUN apt-get install -y nodejs python-gdal p7zip-full && npm install -g nodemon && \
RUN apt-get install -y nodejs p7zip-full && npm install -g nodemon && \
ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine && \
ln -s /code/SuperBuild/install/bin/pdal /usr/bin/pdal

6
MIGRATION.md 100644
Wyświetl plik

@ -0,0 +1,6 @@
# Migration Guide
## From API version 1.x to 2.x
* `skipPostProcessing` parameter in `/task/new` and `/task/new/init` no longer processes 2D tiles via gdal2tiles.py. So the `orthophoto_tiles`, `dsm_tiles` and `dtm_tiles` directories are no longer being generated by NodeODM. Engines can provide such folders if needed and will be included in the output archive.
* `/task/<uuid>/download/orthophoto.tif` has been removed. One can extract the `orthophoto.tif` file from the `all.zip` archive (via `/task/<uuid>/download/all.zip`).

Wyświetl plik

@ -34,6 +34,8 @@ If the computer running NodeODM is using an old or 32bit CPU, you need to compil
See the [API documentation page](https://github.com/OpenDroneMap/NodeODM/blob/master/docs/index.adoc).
Some minor breaking changes exist from version `1.x` to `2.x` of the API. See [migration notes](https://github.com/OpenDroneMap/NodeODM/blob/master/MIGRATION.md).
## Run Tasks from the Command Line
You can use [CloudODM](https://github.com/OpenDroneMap/CloudODM) to run tasks with NodeODM from the command line.

Wyświetl plik

@ -31,7 +31,7 @@ Options:
-p, --port <number> Port to bind the server to (default: 3000)
--odm_path <path> Path to OpenDroneMap's code (default: /code)
--log_level <logLevel> Set log level verbosity (default: info)
-d, --deamonize Set process to run as a deamon
-d, --daemon Set process to run as a deamon
-q, --parallel_queue_processing <number> Number of simultaneous processing tasks (default: 2)
--cleanup_tasks_after <number> Number of minutes that elapse before deleting finished and canceled tasks (default: 2880)
--cleanup_uploads_after <number> Number of minutes that elapse before deleting unfinished uploads. Set this value to the maximum time you expect a dataset to be uploaded. (default: 2880)
@ -94,7 +94,7 @@ config.logger.maxFiles = fromConfigFile("logger.maxFiles", 10); // Max number of
config.logger.logDirectory = fromConfigFile("logger.logDirectory", ''); // Set this to a full path to a directory - if not set logs will be written to the application directory.
config.port = parseInt(argv.port || argv.p || fromConfigFile("port", process.env.PORT || 3000));
config.deamon = argv.deamonize || argv.d || fromConfigFile("daemon", false);
config.deamon = argv.deamonize || argv.daemon || argv.d || fromConfigFile("daemon", false);
config.parallelQueueProcessing = parseInt(argv.parallel_queue_processing || argv.q || fromConfigFile("parallelQueueProcessing", 1));
config.cleanupTasksAfter = parseInt(argv.cleanup_tasks_after || fromConfigFile("cleanupTasksAfter", 2880));
config.cleanupUploadsAfter = parseInt(argv.cleanup_uploads_after || fromConfigFile("cleanupUploadsAfter", 2880));

Wyświetl plik

@ -8,7 +8,7 @@ REST API to access ODM
=== Version information
[%hardbreaks]
_Version_ : 1.6.0
_Version_ : 2.0.0
=== Contact information
@ -348,7 +348,7 @@ _optional_|Serialized JSON string of the options to use for processing, as an ar
|*FormData*|*outputs* +
_optional_|An optional serialized JSON string of paths relative to the project directory that should be included in the all.zip result file, overriding the default behavior.|string|
|*FormData*|*skipPostProcessing* +
_optional_|When set, skips generation of map tiles, derivate assets, point cloud tiles.|boolean|
_optional_|When set, skips generation of point cloud tiles.|boolean|
|*FormData*|*webhook* +
_optional_|Optional URL to call when processing has ended (either successfully or unsuccessfully).|string|
|*FormData*|*zipurl* +
@ -455,7 +455,7 @@ _optional_|Serialized JSON string of the options to use for processing, as an ar
|*FormData*|*outputs* +
_optional_|An optional serialized JSON string of paths relative to the project directory that should be included in the all.zip result file, overriding the default behavior.|string|
|*FormData*|*skipPostProcessing* +
_optional_|When set, skips generation of map tiles, derivate assets, point cloud tiles.|boolean|
_optional_|When set, skips generation of point cloud tiles.|boolean|
|*FormData*|*webhook* +
_optional_|Optional URL to call when processing has ended (either successfully or unsuccessfully).|string|
|===
@ -598,7 +598,7 @@ Retrieves an asset (the output of OpenDroneMap's processing) associated with a t
|===
|Type|Name|Description|Schema|Default
|*Path*|*asset* +
_required_|Type of asset to download. Use "all.zip" for zip file containing all assets.|enum (all.zip, orthophoto.tif)|
_required_|Type of asset to download. Use "all.zip" for zip file containing all assets.|enum (all.zip)|
|*Path*|*uuid* +
_required_|UUID of the task|string|
|*Query*|*token* +

File diff suppressed because one or more lines are too long

Wyświetl plik

@ -71,7 +71,7 @@ let server;
* -
* name: skipPostProcessing
* in: formData
* description: 'When set, skips generation of map tiles, derivate assets, point cloud tiles.'
* description: 'When set, skips generation of point cloud tiles.'
* required: false
* type: boolean
* -
@ -229,7 +229,7 @@ app.post('/task/new/commit/:uuid', authCheck, taskNew.getUUID, taskNew.handleCom
* -
* name: skipPostProcessing
* in: formData
* description: 'When set, skips generation of map tiles, derivate assets, point cloud tiles.'
* description: 'When set, skips generation of point cloud tiles.'
* required: false
* type: boolean
* -
@ -476,7 +476,6 @@ app.get('/task/:uuid/output', authCheck, getTaskFromUuid, (req, res) => {
* required: true
* enum:
* - all.zip
* - orthophoto.tif
* -
* name: token
* in: query

Wyświetl plik

@ -137,13 +137,6 @@ module.exports = class Task{
getAssetsArchivePath(filename){
if (filename == 'all.zip'){
// OK, do nothing
}else if (filename == 'orthophoto.tif'){
if (config.test){
if (config.testSkipOrthophotos) return false;
else filename = path.join('..', '..', 'processing_results', 'odm_orthophoto', `odm_${filename}`);
}else{
filename = path.join('odm_orthophoto', `odm_${filename}`);
}
}else{
return false; // Invalid
}
@ -435,7 +428,7 @@ module.exports = class Task{
}else if (config.s3UploadEverything){
s3Paths = ['all.zip'].concat(allPaths);
}else{
s3Paths = ['all.zip', 'odm_orthophoto/odm_orthophoto.tif'];
s3Paths = ['all.zip'];
}
S3.uploadPaths(this.getProjectFolderPath(), config.s3Bucket, this.uuid, s3Paths,

Wyświetl plik

@ -38,22 +38,19 @@ try {
logPath += path.sep;
logPath += config.instance + ".log";
let logger = new (winston.Logger)({
transports: [
new (winston.transports.Console)({ level: config.logger.level }),
]
});
logger.add(winston.transports.File, {
let transports = [];
if (!config.deamon){
transports.push(new winston.transports.Console({ level: config.logger.level, format: winston.format.simple() }));
}
let logger = winston.createLogger({ transports });
logger.add(new winston.transports.File({
format: winston.format.simple(),
filename: logPath, // Write to projectname.log
json: false, // Write in plain text, not JSON
maxsize: config.logger.maxFileSize, // Max size of each file
maxFiles: config.logger.maxFiles, // Max number of files
level: config.logger.level // Level of log messages
});
if (config.deamon){
// Console transport is no use to us when running as a daemon
logger.remove(winston.transports.Console);
}
}));
module.exports = logger;

Wyświetl plik

@ -112,38 +112,46 @@ module.exports = {
return; // Skip rest
}
// Launch
const env = utils.clone(process.env);
env.ODM_OPTIONS_TMP_FILE = utils.tmpPath(".json");
let childProcess = spawn("python", [path.join(__dirname, "..", "helpers", "odmOptionsToJson.py"),
"--project-path", config.odm_path, "bogusname"], { env });
// Cleanup on done
let handleResult = (err, result) => {
fs.exists(env.ODM_OPTIONS_TMP_FILE, exists => {
if (exists) fs.unlink(env.ODM_OPTIONS_TMP_FILE, err => {
if (err) console.warning(`Cannot cleanup ${env.ODM_OPTIONS_TMP_FILE}`);
});
});
// Don't wait
done(err, result);
};
childProcess
.on('exit', (code, signal) => {
try{
fs.readFile(env.ODM_OPTIONS_TMP_FILE, { encoding: "utf8" }, (err, data) => {
if (err) handleResult(new Error(`Cannot read list of options from ODM (from temporary file). Is ODM installed in ${config.odm_path}?`));
else{
let json = JSON.parse(data);
handleResult(null, json);
}
const getOdmOptions = (pythonExe, done) => {
// Launch
const env = utils.clone(process.env);
env.ODM_OPTIONS_TMP_FILE = utils.tmpPath(".json");
let childProcess = spawn(pythonExe, [path.join(__dirname, "..", "helpers", "odmOptionsToJson.py"),
"--project-path", config.odm_path, "bogusname"], { env });
// Cleanup on done
let handleResult = (err, result) => {
fs.exists(env.ODM_OPTIONS_TMP_FILE, exists => {
if (exists) fs.unlink(env.ODM_OPTIONS_TMP_FILE, err => {
if (err) console.warning(`Cannot cleanup ${env.ODM_OPTIONS_TMP_FILE}`);
});
}catch(err){
handleResult(new Error(`Could not load list of options from ODM. Is ODM installed in ${config.odm_path}? Make sure that OpenDroneMap is installed and that --odm_path is set properly: ${err.message}`));
}
})
.on('error', handleResult);
});
// Don't wait
done(err, result);
};
childProcess
.on('exit', (code, signal) => {
try{
fs.readFile(env.ODM_OPTIONS_TMP_FILE, { encoding: "utf8" }, (err, data) => {
if (err) handleResult(new Error(`Cannot read list of options from ODM (from temporary file). Is ODM installed in ${config.odm_path}?`));
else{
let json = JSON.parse(data);
handleResult(null, json);
}
});
}catch(err){
handleResult(new Error(`Could not load list of options from ODM. Is ODM installed in ${config.odm_path}? Make sure that OpenDroneMap is installed and that --odm_path is set properly: ${err.message}`));
}
})
.on('error', handleResult);
}
// Try Python3 first
getOdmOptions("python3", (err, result) => {
if (err) getOdmOptions("python", done);
else done(null, result);
});
}
};

Wyświetl plik

@ -1,6 +1,6 @@
{
"name": "NodeODM",
"version": "1.6.1",
"version": "2.0.0",
"description": "REST API to access ODM",
"main": "index.js",
"scripts": {
@ -38,7 +38,7 @@
"systeminformation": "3.42.0",
"tree-kill": "^1.2.1",
"uuid": "^3.3.2",
"winston": "^2.2.0"
"winston": "^3.3.3"
},
"devDependencies": {
"grunt": "^1.0.3",

Wyświetl plik

@ -466,15 +466,9 @@ $(function() {
Task.prototype.downloadLink = function(){
return "/task/" + this.uuid + "/download/all.zip?token=" + token;
};
Task.prototype.downloadOrthoLink = function(){
return "/task/" + this.uuid + "/download/orthophoto.tif?token=" + token;
};
Task.prototype.download = function() {
location.href = this.downloadLink();
};
Task.prototype.downloadOrthophoto = function() {
location.href = this.downloadOrthoLink();
};
var taskList = new TaskList();
ko.applyBindings(taskList, document.getElementById('taskList'));

Wyświetl plik

@ -1,12 +0,0 @@
0% 255 0 255
10% 128 0 255
20% 0 0 255
30% 0 128 255
40% 0 255 255
50% 0 255 128
60% 0 255 0
70% 128 255 0
80% 255 255 0
90% 255 128 0
100% 255 0 0
nv 0 0 0 0

Plik diff jest za duży Load Diff

Wyświetl plik

@ -1,234 +0,0 @@
#!/usr/bin/env python
#******************************************************************************
# $Id$
#
# Project: GDAL Python Interface
# Purpose: Script to merge greyscale as intensity into an RGB(A) image, for
# instance to apply hillshading to a dem colour relief.
# Author: Frank Warmerdam, warmerdam@pobox.com
# Trent Hare (USGS)
#
#******************************************************************************
# Copyright (c) 2009, Frank Warmerdam
# Copyright (c) 2010, Even Rouault <even dot rouault at mines-paris dot org>
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
#******************************************************************************
import sys
import numpy
from osgeo import gdal
# =============================================================================
# rgb_to_hsv()
#
# rgb comes in as [r,g,b] with values in the range [0,255]. The returned
# hsv values will be with hue and saturation in the range [0,1] and value
# in the range [0,255]
#
def rgb_to_hsv( r,g,b ):
maxc = numpy.maximum(r,numpy.maximum(g,b))
minc = numpy.minimum(r,numpy.minimum(g,b))
v = maxc
minc_eq_maxc = numpy.equal(minc,maxc)
# compute the difference, but reset zeros to ones to avoid divide by zeros later.
ones = numpy.ones((r.shape[0],r.shape[1]))
maxc_minus_minc = numpy.choose( minc_eq_maxc, (maxc-minc,ones) )
s = (maxc-minc) / numpy.maximum(ones,maxc)
rc = (maxc-r) / maxc_minus_minc
gc = (maxc-g) / maxc_minus_minc
bc = (maxc-b) / maxc_minus_minc
maxc_is_r = numpy.equal(maxc,r)
maxc_is_g = numpy.equal(maxc,g)
maxc_is_b = numpy.equal(maxc,b)
h = numpy.zeros((r.shape[0],r.shape[1]))
h = numpy.choose( maxc_is_b, (h,4.0+gc-rc) )
h = numpy.choose( maxc_is_g, (h,2.0+rc-bc) )
h = numpy.choose( maxc_is_r, (h,bc-gc) )
h = numpy.mod(h/6.0,1.0)
hsv = numpy.asarray([h,s,v])
return hsv
# =============================================================================
# hsv_to_rgb()
#
# hsv comes in as [h,s,v] with hue and saturation in the range [0,1],
# but value in the range [0,255].
def hsv_to_rgb( hsv ):
h = hsv[0]
s = hsv[1]
v = hsv[2]
#if s == 0.0: return v, v, v
i = (h*6.0).astype(int)
f = (h*6.0) - i
p = v*(1.0 - s)
q = v*(1.0 - s*f)
t = v*(1.0 - s*(1.0-f))
r = i.choose( v, q, p, p, t, v )
g = i.choose( t, v, v, q, p, p )
b = i.choose( p, p, t, v, v, q )
rgb = numpy.asarray([r,g,b]).astype(numpy.uint8)
return rgb
# =============================================================================
# Usage()
def Usage():
print("""Usage: hsv_merge.py [-q] [-of format] src_color src_greyscale dst_color
where src_color is a RGB or RGBA dataset,
src_greyscale is a greyscale dataset (e.g. the result of gdaldem hillshade)
dst_color will be a RGB or RGBA dataset using the greyscale as the
intensity for the color dataset.
""")
sys.exit(1)
# =============================================================================
# Mainline
# =============================================================================
argv = gdal.GeneralCmdLineProcessor( sys.argv )
if argv is None:
sys.exit( 0 )
format = 'GTiff'
src_color_filename = None
src_greyscale_filename = None
dst_color_filename = None
quiet = False
# Parse command line arguments.
i = 1
while i < len(argv):
arg = argv[i]
if arg == '-of':
i = i + 1
format = argv[i]
elif arg == '-q' or arg == '-quiet':
quiet = True
elif src_color_filename is None:
src_color_filename = argv[i]
elif src_greyscale_filename is None:
src_greyscale_filename = argv[i]
elif dst_color_filename is None:
dst_color_filename = argv[i]
else:
Usage()
i = i + 1
if dst_color_filename is None:
Usage()
datatype = gdal.GDT_Byte
hilldataset = gdal.Open( src_greyscale_filename, gdal.GA_ReadOnly )
colordataset = gdal.Open( src_color_filename, gdal.GA_ReadOnly )
#check for 3 or 4 bands in the color file
if (colordataset.RasterCount != 3 and colordataset.RasterCount != 4):
print('Source image does not appear to have three or four bands as required.')
sys.exit(1)
#define output format, name, size, type and set projection
out_driver = gdal.GetDriverByName(format)
outdataset = out_driver.Create(dst_color_filename, colordataset.RasterXSize, \
colordataset.RasterYSize, colordataset.RasterCount, datatype)
outdataset.SetProjection(hilldataset.GetProjection())
outdataset.SetGeoTransform(hilldataset.GetGeoTransform())
#assign RGB and hillshade bands
rBand = colordataset.GetRasterBand(1)
gBand = colordataset.GetRasterBand(2)
bBand = colordataset.GetRasterBand(3)
if colordataset.RasterCount == 4:
aBand = colordataset.GetRasterBand(4)
else:
aBand = None
hillband = hilldataset.GetRasterBand(1)
hillbandnodatavalue = hillband.GetNoDataValue()
#check for same file size
if ((rBand.YSize != hillband.YSize) or (rBand.XSize != hillband.XSize)):
print('Color and hillshade must be the same size in pixels.')
sys.exit(1)
#loop over lines to apply hillshade
for i in range(hillband.YSize):
#load RGB and Hillshade arrays
rScanline = rBand.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
gScanline = gBand.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
bScanline = bBand.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
hillScanline = hillband.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
#convert to HSV
hsv = rgb_to_hsv( rScanline, gScanline, bScanline )
# if there's nodata on the hillband, use the v value from the color
# dataset instead of the hillshade value.
if hillbandnodatavalue is not None:
equal_to_nodata = numpy.equal(hillScanline, hillbandnodatavalue)
v = numpy.choose(equal_to_nodata,(hillScanline,hsv[2]))
else:
v = hillScanline
#replace v with hillshade
hsv_adjusted = numpy.asarray( [hsv[0], hsv[1], v] )
#convert back to RGB
dst_color = hsv_to_rgb( hsv_adjusted )
#write out new RGB bands to output one band at a time
outband = outdataset.GetRasterBand(1)
outband.WriteArray(dst_color[0], 0, i)
outband = outdataset.GetRasterBand(2)
outband.WriteArray(dst_color[1], 0, i)
outband = outdataset.GetRasterBand(3)
outband.WriteArray(dst_color[2], 0, i)
if aBand is not None:
aScanline = aBand.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
outband = outdataset.GetRasterBand(4)
outband.WriteArray(aScanline, 0, i)
#update progress line
if not quiet:
gdal.TermProgress_nocb( (float(i+1) / hillband.YSize) )

Wyświetl plik

@ -19,43 +19,6 @@ script_path=$(realpath $(dirname "$0"))
cd "$script_path/../$1"
echo "Postprocessing: $(pwd)"
# Generate colored shaded relief for DTM/DSM files if available
dem_products=()
if [ -e "odm_dem/dsm.tif" ]; then dem_products=(${dem_products[@]} dsm); fi
if [ -e "odm_dem/dtm.tif" ]; then dem_products=(${dem_products[@]} dtm); fi
if hash gdaldem 2>/dev/null; then
for dem_product in ${dem_products[@]}; do
dem_path="odm_dem/""$dem_product"".tif"
gdaldem color-relief $dem_path $script_path/color_relief.txt "odm_dem/""$dem_product""_colored.tif" -alpha -co ALPHA=YES
gdaldem hillshade $dem_path "odm_dem/""$dem_product""_hillshade.tif" -z 1.0 -s 1.0 -az 315.0 -alt 45.0
python "$script_path/hsv_merge.py" "odm_dem/""$dem_product""_colored.tif" "odm_dem/""$dem_product""_hillshade.tif" "odm_dem/""$dem_product""_colored_hillshade.tif"
done
else
echo "gdaldem is not installed, will skip colored hillshade generation"
fi
# Generate Tiles
g2t_options="--processes $(nproc) -z 5-21 -n -w none"
orthophoto_path="odm_orthophoto/odm_orthophoto.tif"
if [ -e "$orthophoto_path" ]; then
python "$script_path/gdal2tiles.py" $g2t_options $orthophoto_path orthophoto_tiles
# Check for DEM tiles also
for dem_product in ${dem_products[@]}; do
colored_dem_path="odm_dem/""$dem_product""_colored_hillshade.tif"
if [ -e "$colored_dem_path" ]; then
python "$script_path/gdal2tiles.py" $g2t_options $colored_dem_path "$dem_product""_tiles"
else
echo "No $dem_product found at $colored_dem_path: will skip tiling"
fi
done
else
echo "No orthophoto found at $orthophoto_path: will skip tiling"
fi
# Generate point cloud (if entwine or potreeconverter is available)
pointcloud_input_path=""
for path in "odm_georeferencing/odm_georeferenced_model.laz" \