mirror
/
OpenDroneMap-NodeODM
kopia lustrzana https://github.com/OpenDroneMap/NodeODM
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność

Merge branch 'master' into swagger

pull/1/head
Matthew Berryman 2016-07-31 14:03:54 +10:00
rodzic 047517460a 14d0ac17a1
commit 19f4c87da2
17 zmienionych plików z 719 dodań i 176 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

15
README.md
Wyświetl plik

@ -1,11 +1,15 @@
# Open Source Drone Aerial Imagery Processing
node-OpenDroneMap is a Node.js App and REST API to access [OpenDroneMap](https://github.com/OpenDroneMap/OpenDroneMap)
[http://nodeodm.masseranolabs.com](http://nodeodm.masseranolabs.com)
![Alt text](/screenshots/main.png?raw=true "Node-OpenDroneMap")
## Getting Started
The quickest way is to use [Docker](https://www.docker.com/).
For a quick taste of the application, we have setup a test environment at [http://nodeodm.masseranolabs.com](http://nodeodm.masseranolabs.com). Please note that **this is not a production environment**, and that processing on this server will be slow (you are sharing the server's resources with everyone else in the world).
If you want to do your own imagery processing, we recommend that you setup your own instance via [Docker](https://www.docker.com/).
* From the Docker Quickstart Terminal (Windows / OSX) or from the command line (Linux) type:
```
@ -47,12 +51,15 @@ You can find some test drone images from [OpenDroneMap's Test Data Folder](https
## Contributing
Make a pull request for small contributions. For big contributions, please open a discussion first.
Make a pull request to the dev branch for small contributions. For big contributions, please open a discussion first. Please use ES6 syntax while writing new Javascript code so that we can keep the code base uniform.
## Roadmap
- [ ] Command line options for OpenDroneMap (in progress)
- [ ] Cluster tasks distribution to multiple servers
- [X] Command line options for OpenDroneMap
- [X] GPC List support
- [ ] Autoremove Abandoned Tasks
- [ ] Queue Position Tracking
- [ ] Continuous Integration Setup
- [ ] Documentation
- [ ] Unit Testing

62
config.js
Wyświetl plik

@ -1,35 +1,57 @@
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
'use strict';
// config.js - Configuration for cognicity-server
let argv = require('minimist')(process.argv.slice(2));
if (argv.help){
console.log(`
Usage: node index.js [options]
/**
* Cognicity server configuration object.
* @namespace {object} config
* @property {string} instance The name of this instance of the server
* @property {object} logger Configuration options for logging
* @property {string} logger.level Log level - info, verbose or debug are most useful. Levels are (npm defaults): silly, debug, verbose, info, warn, error.
* @property {number} logger.maxFileSize Maximum size of each log file in bytes
* @property {number} logger.maxFiles Maximum number of log files to keep
* @property {?number} logger.logDirectory Full path to directory to store log files in, if not set logs will be written to the application directory
* @property {number} port Port to launch server on
*/
Options:
-p, --port <number> Port to bind the server to (default: 3000)
--odm_path <path> Path to OpenDroneMap's code (default: /code)
--log_level <logLevel> Set log level verbosity (default: info)
-d, --deamonize Set process to run as a deamon
--parallel_queue_processing <number> Number of simultaneous processing tasks (default: 2)
--cleanup_tasks_after <number> Number of days that elapse before deleting finished and canceled tasks (default: 3)
var config = {};
Log Levels:
error | debug | info | verbose | debug | silly
`);
process.exit(0);
}
// Instance name - default name for this configuration (will be server process name)
let config = {};
// Instance name - default name for this configuration
config.instance = 'node-OpenDroneMap';
config.odm_path = argv.odm_path || '/code';
// Logging configuration
config.logger = {};
config.logger.level = "debug"; // What level to log at; info, verbose or debug are most useful. Levels are (npm defaults): silly, debug, verbose, info, warn, error.
config.logger.level = argv.log_level || 'info'; // What level to log at; info, verbose or debug are most useful. Levels are (npm defaults): silly, debug, verbose, info, warn, error.
config.logger.maxFileSize = 1024 * 1024 * 100; // Max file size in bytes of each log file; default 100MB
config.logger.maxFiles = 10; // Max number of log files kept
config.logger.logDirectory = ''; // Set this to a full path to a directory - if not set logs will be written to the application directory.
// Server port
config.port = process.env.PORT || 8081;
// process.env.PORT is what AWS Elastic Beanstalk defines
// on IBM bluemix use config.port = process.env.VCAP_APP_PORT || 8081;
config.port = parseInt(argv.port || argv.p || process.env.PORT || 3000);
config.deamon = argv.deamonize || argv.d;
config.parallelQueueProcessing = argv.parallel_queue_processing || 2;
config.cleanupTasksAfter = argv.cleanup_tasks_after || 3;
module.exports = config;

38
helpers/odmOptionsToJson.py 100644
Wyświetl plik

@ -0,0 +1,38 @@
#!/usr/bin/env python
'''
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
'''
import sys
import imp
import argparse
import json
imp.load_source('context', sys.argv[2] + '/opendm/context.py')
odm = imp.load_source('config', sys.argv[2] + '/opendm/config.py')
options = {}
class ArgumentParserStub(argparse.ArgumentParser):
def add_argument(self, *args, **kwargs):
argparse.ArgumentParser.add_argument(self, *args, **kwargs)
options[args[0]] = {}
for name, value in kwargs.items():
options[args[0]][str(name)] = str(value)
odm.parser = ArgumentParserStub()
odm.config()
print json.dumps(options)

106
index.js
Wyświetl plik

@ -17,9 +17,9 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
"use strict";
var config = require('./config.js')
let config = require('./config.js');
let logger = require('winston');
let logger = require('./libs/logger');
let fs = require('fs');
let path = require('path');
let async = require('async');
@ -32,39 +32,15 @@ let multer = require('multer');
let bodyParser = require('body-parser');
let morgan = require('morgan');
// Set up logging
// Configure custom File transport to write plain text messages
var logPath = ( config.logger.logDirectory ? config.logger.logDirectory : __dirname );
// Check that log file directory can be written to
try {
fs.accessSync(logPath, fs.W_OK);
} catch (e) {
console.log( "Log directory '" + logPath + "' cannot be written to" );
throw e;
}
logPath += path.sep;
logPath += config.instance + ".log";
let TaskManager = require('./libs/TaskManager');
let Task = require('./libs/Task');
let odmOptions = require('./libs/odmOptions');
logger
.add(logger.transports.File, {
filename: logPath, // Write to projectname.log
json: false, // Write in plain text, not JSON
maxsize: config.logger.maxFileSize, // Max size of each file
maxFiles: config.logger.maxFiles, // Max number of files
level: config.logger.level // Level of log messages
})
// Console transport is no use to us when running as a daemon
.remove(logger.transports.Console);
var winstonStream = {
let winstonStream = {
write: function(message, encoding){
logger.info(message.slice(0, -1));
logger.debug(message.slice(0, -1));
}
};
let TaskManager = require('./libs/taskManager');
let Task = require('./libs/Task');
app.use(morgan('combined', { stream : winstonStream }));
app.use(bodyParser.urlencoded({extended: true}));
app.use(bodyParser.json());
@ -93,30 +69,56 @@ let upload = multer({
app.post('/task/new', addRequestId, upload.array('images'), (req, res) => {
if (req.files.length === 0) res.json({error: "Need at least 1 file."});
else{
// Move to data
let srcPath = `tmp/${req.id}`;
let destPath = `data/${req.id}`;
let destImagesPath = `${destPath}/images`;
let destGpcPath = `${destPath}/gpc`;
async.series([
cb => {
fs.stat(`data/${req.id}`, (err, stat) => {
odmOptions.filterOptions(req.body.options, (err, options) => {
if (err) cb(err);
else{
req.body.options = options;
cb(null);
}
});
},
// Move all uploads to data/<uuid>/images dir
cb => {
fs.stat(destPath, (err, stat) => {
if (err && err.code === 'ENOENT') cb();
else cb(new Error(`Directory exists (should not have happened: ${err.code})`));
});
},
cb => { fs.mkdir(`data/${req.id}`, undefined, cb); },
cb => fs.mkdir(destPath, undefined, cb),
cb => fs.mkdir(destGpcPath, undefined, cb),
cb => fs.rename(srcPath, destImagesPath, cb),
cb => {
fs.rename(`tmp/${req.id}`, `data/${req.id}/images`, err => {
if (!err){
new Task(req.id, req.body.name, (err, task) => {
if (err) cb(err);
else{
taskManager.addNew(task);
res.json({uuid: req.id, success: true});
cb();
}
});
}else{
cb(new Error("Could not move images folder."))
// Find any *.txt (GPC) file and move it to the data/<uuid>/gpc directory
fs.readdir(destImagesPath, (err, entries) => {
if (err) cb(err);
else{
async.eachSeries(entries, (entry, cb) => {
if (/\.txt$/gi.test(entry)){
fs.rename(path.join(destImagesPath, entry), path.join(destGpcPath, entry), cb);
}else cb();
}, cb);
}
});
},
// Create task
cb => {
new Task(req.id, req.body.name, (err, task) => {
if (err) cb(err);
else{
taskManager.addNew(task);
res.json({uuid: req.id, success: true});
cb();
}
}, req.body.options);
}
], err => {
if (err) res.json({error: err.message})
@ -184,9 +186,16 @@ app.post('/task/restart', uuidCheck, (req, res) => {
taskManager.restart(req.body.uuid, successHandler(res));
});
app.get('/getOptions', (req, res) => {
odmOptions.getOptions((err, options) => {
if (err) res.json({error: err.message});
else res.json(options);
});
});
let gracefulShutdown = done => {
async.series([
cb => { taskManager.dumpTaskList(cb) },
cb => taskManager.dumpTaskList(cb),
cb => {
logger.info("Closing server");
server.close();
@ -207,7 +216,8 @@ let taskManager;
let server;
async.series([
cb => { taskManager = new TaskManager(cb,logger); },
cb => odmOptions.initialize(cb),
cb => { taskManager = new TaskManager(cb); },
cb => { server = app.listen(config.port, err => {
if (!err) logger.info('Server has started on port ' + String(config.port));
cb(err);

89
libs/Task.js
Wyświetl plik

@ -1,5 +1,5 @@
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
@ -16,16 +16,21 @@ You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
"use strict";
let async = require('async');
let assert = require('assert');
let logger = require('./logger');
let fs = require('fs');
let path = require('path');
let rmdir = require('rimraf');
let odmRunner = require('./odmRunner');
let archiver = require('archiver');
let os = require('os');
let statusCodes = require('./statusCodes');
module.exports = class Task{
constructor(uuid, name, done){
constructor(uuid, name, done, options = []){
assert(uuid !== undefined, "uuid must be set");
assert(done !== undefined, "ready must be set");
@ -34,17 +39,41 @@ module.exports = class Task{
this.dateCreated = new Date().getTime();
this.processingTime = -1;
this.setStatus(statusCodes.QUEUED);
this.options = {};
this.options = options;
this.gpcFiles = [];
this.output = [];
this.runnerProcess = null;
// Read images info
fs.readdir(this.getImagesFolderPath(), (err, files) => {
if (err) done(err);
else{
this.images = files;
done(null, this);
async.series([
// Read images info
cb => {
fs.readdir(this.getImagesFolderPath(), (err, files) => {
if (err) cb(err);
else{
this.images = files;
logger.debug(`Found ${this.images.length} images for ${this.uuid}`)
cb(null);
}
});
},
// Find GCP (if any)
cb => {
fs.readdir(this.getGpcFolderPath(), (err, files) => {
if (err) cb(err);
else{
files.forEach(file => {
if (/\.txt$/gi.test(file)){
this.gpcFiles.push(file);
}
});
logger.debug(`Found ${this.gpcFiles.length} GPC files (${this.gpcFiles.join(" ")}) for ${this.uuid}`);
cb(null);
}
});
}
], err => {
done(err, this);
});
}
@ -57,14 +86,14 @@ module.exports = class Task{
for (let k in taskJson){
task[k] = taskJson[k];
}
// Tasks that were running should be put back to QUEUED state
if (task.status.code === statusCodes.RUNNING){
task.status.code = statusCodes.QUEUED;
}
done(null, task);
}
})
}, taskJson.options);
}
// Get path where images are stored for this task
@ -73,6 +102,12 @@ module.exports = class Task{
return `${this.getProjectFolderPath()}/images`;
}
// Get path where GPC file(s) are stored
// (relative to nodejs process CWD)
getGpcFolderPath(){
return `${this.getProjectFolderPath()}/gpc`;
}
// Get path of project (where all images and assets folder are contained)
// (relative to nodejs process CWD)
getProjectFolderPath(){
@ -94,13 +129,13 @@ module.exports = class Task{
this.status = {
code: code
};
for (var k in extra){
for (let k in extra){
this.status[k] = extra[k];
}
}
updateProcessingTime(resetTime){
this.processingTime = resetTime ?
this.processingTime = resetTime ?
-1 :
new Date().getTime() - this.dateCreated;
}
@ -135,12 +170,12 @@ module.exports = class Task{
if (this.status.code !== statusCodes.CANCELED){
let wasRunning = this.status.code === statusCodes.RUNNING;
this.setStatus(statusCodes.CANCELED);
if (wasRunning && this.runnerProcess){
// TODO: this does guarantee that
// TODO: this does NOT guarantee that
// the process will immediately terminate.
// In fact, often times ODM will continue running for a while
// This might need to be fixed on ODM's end.
// This might need to be fixed on ODM's end.
this.runnerProcess.kill('SIGINT');
this.runnerProcess = null;
}
@ -187,9 +222,21 @@ module.exports = class Task{
if (this.status.code === statusCodes.QUEUED){
this.startTrackingProcessingTime();
this.setStatus(statusCodes.RUNNING);
this.runnerProcess = odmRunner.run({
projectPath: `${__dirname}/../${this.getProjectFolderPath()}`
}, (err, code, signal) => {
let runnerOptions = this.options.reduce((result, opt) => {
result[opt.name] = opt.value;
return result;
}, {});
runnerOptions["project-path"] = fs.realpathSync(this.getProjectFolderPath());
runnerOptions["pmvs-num-cores"] = os.cpus().length;
if (this.gpcFiles.length > 0){
runnerOptions["odm_georeferencing-useGcp"] = true;
runnerOptions["odm_georeferencing-gcpFile"] = fs.realpathSync(path.join(this.getGpcFolderPath(), this.gpcFiles[0]));
}
this.runnerProcess = odmRunner.run(runnerOptions, (err, code, signal) => {
if (err){
this.setStatus(statusCodes.FAILED, {errorMessage: `Could not start process (${err.message})`});
finished();
@ -262,4 +309,4 @@ module.exports = class Task{
options: this.options
}
}
};
};

64
libs/TaskManager.js
Wyświetl plik

@ -17,32 +17,36 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
"use strict";
let assert = require('assert');
let config = require('../config');
let rmdir = require('rimraf');
let fs = require('fs');
let path = require('path');
let logger = require('./logger');
let Task = require('./Task');
let statusCodes = require('./statusCodes');
let async = require('async');
let schedule = require('node-schedule');
const PARALLEL_QUEUE_PROCESS_LIMIT = 2;
const TASKS_DUMP_FILE = "data/tasks.json";
const CLEANUP_TASKS_IF_OLDER_THAN = 1000 * 60 * 60 * 24 * 3; // 3 days
const DATA_DIR = "data";
const TASKS_DUMP_FILE = `${DATA_DIR}/tasks.json`;
const CLEANUP_TASKS_IF_OLDER_THAN = 1000 * 60 * 60 * 24 * config.cleanupTasksAfter; // days
module.exports = class TaskManager{
constructor(done,logger){
this.logger = logger;
constructor(done){
this.tasks = {};
this.runningQueue = [];
async.series([
cb => { this.restoreTaskListFromDump(cb); },
cb => { this.removeOldTasks(cb); },
cb => this.restoreTaskListFromDump(cb),
cb => this.removeOldTasks(cb),
cb => this.removeOrphanedDirectories(cb),
cb => {
this.processNextTask();
cb();
},
cb => {
// Every hour
schedule.scheduleJob('* 0 * * * *', () => {
schedule.scheduleJob('0 * * * *', () => {
this.removeOldTasks();
this.dumpTaskList();
});
@ -57,7 +61,7 @@ module.exports = class TaskManager{
removeOldTasks(done){
let list = [];
let now = new Date().getTime();
this.logger.info("Checking for old tasks to be removed...");
logger.info("Checking for old tasks to be removed...");
for (let uuid in this.tasks){
let task = this.tasks[uuid];
@ -71,11 +75,32 @@ module.exports = class TaskManager{
}
async.eachSeries(list, (uuid, cb) => {
this.logger.info(`Cleaning up old task ${uuid}`)
logger.info(`Cleaning up old task ${uuid}`)
this.remove(uuid, cb);
}, done);
}
// Removes directories that don't have a corresponding
// task associated with it (maybe as a cause of an abrupt exit)
removeOrphanedDirectories(done){
logger.info("Checking for orphaned directories to be removed...");
fs.readdir(DATA_DIR, (err, entries) => {
if (err) done(err);
else{
async.eachSeries(entries, (entry, cb) => {
let dirPath = path.join(DATA_DIR, entry);
if (fs.statSync(dirPath).isDirectory() &&
entry.match(/^[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+$/) &&
!this.tasks[entry]){
logger.info(`Found orphaned directory: ${entry}, removing...`);
rmdir(dirPath, cb);
}else cb();
}, done);
}
});
}
// Load tasks that already exists (if any)
restoreTaskListFromDump(done){
fs.readFile(TASKS_DUMP_FILE, (err, data) => {
@ -91,11 +116,11 @@ module.exports = class TaskManager{
}
});
}, err => {
this.logger.info(`Initialized ${tasks.length} tasks`);
logger.info(`Initialized ${tasks.length} tasks`);
if (done !== undefined) done();
});
}else{
this.logger.info("No tasks dump found");
logger.info("No tasks dump found");
if (done !== undefined) done();
}
});
@ -113,7 +138,7 @@ module.exports = class TaskManager{
// Finds the next tasks, adds them to the running queue,
// and starts the tasks (up to the limit).
processNextTask(){
if (this.runningQueue.length < PARALLEL_QUEUE_PROCESS_LIMIT){
if (this.runningQueue.length < config.parallelQueueProcessing){
let task = this.findNextTaskToProcess();
if (task){
this.addToRunningQueue(task);
@ -122,7 +147,7 @@ module.exports = class TaskManager{
this.processNextTask();
});
if (this.runningQueue.length < PARALLEL_QUEUE_PROCESS_LIMIT) this.processNextTask();
if (this.runningQueue.length < config.parallelQueueProcessing) this.processNextTask();
}
}else{
// Do nothing
@ -136,10 +161,7 @@ module.exports = class TaskManager{
removeFromRunningQueue(task){
assert(task.constructor.name === "Task", "Must be a Task object");
this.runningQueue = this.runningQueue.filter(t => {
return t !== task;
});
this.runningQueue = this.runningQueue.filter(t => t !== task);
}
addNew(task){
@ -207,15 +229,15 @@ module.exports = class TaskManager{
// Serializes the list of tasks and saves it
// to disk
dumpTaskList(done){
var output = [];
let output = [];
for (let uuid in this.tasks){
output.push(this.tasks[uuid].serialize());
}
fs.writeFile(TASKS_DUMP_FILE, JSON.stringify(output), err => {
if (err) this.logger.error(`Could not dump tasks: ${err.message}`);
else this.logger.debug("Dumped tasks list.");
if (err) logger.error(`Could not dump tasks: ${err.message}`);
else logger.info("Dumped tasks list.");
if (done !== undefined) done();
})
}

4
libs/expressRequestId.js
Wyświetl plik

@ -1,6 +1,6 @@
'use strict';
var uuid = require('node-uuid');
let uuid = require('node-uuid');
module.exports = function (options) {
options = options || {};
@ -14,4 +14,4 @@ module.exports = function (options) {
}
next();
};
};
};

59
libs/logger.js 100644
Wyświetl plik

@ -0,0 +1,59 @@
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
"use strict";
let config = require('../config');
let winston = require('winston');
let fs = require('fs');
let path = require('path');
// Set up logging
// Configure custom File transport to write plain text messages
let logPath = ( config.logger.logDirectory ?
config.logger.logDirectory :
`${__dirname}/../` );
// Check that log file directory can be written to
try {
fs.accessSync(logPath, fs.W_OK);
} catch (e) {
console.log( "Log directory '" + logPath + "' cannot be written to" );
throw e;
}
logPath += path.sep;
logPath += config.instance + ".log";
let logger = new (winston.Logger)({
transports: [
new (winston.transports.Console)({ level: config.logger.level }),
]
});
logger.add(winston.transports.File, {
filename: logPath, // Write to projectname.log
json: false, // Write in plain text, not JSON
maxsize: config.logger.maxFileSize, // Max size of each file
maxFiles: config.logger.maxFiles, // Max number of files
level: config.logger.level // Level of log messages
})
if (config.deamon){
// Console transport is no use to us when running as a daemon
logger.remove(winston.transports.Console);
}
module.exports = logger;

214
libs/odmOptions.js 100644
Wyświetl plik

@ -0,0 +1,214 @@
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
"use strict";
let odmRunner = require('./odmRunner');
let assert = require('assert');
let odmOptions = null;
module.exports = {
initialize: function(done){
this.getOptions(done);
},
getOptions: function(done){
if (odmOptions){
done(null, odmOptions);
return;
}
odmRunner.getJsonOptions((err, json) => {
if (err) done(err);
else{
odmOptions = [];
for (let option in json){
// Not all options are useful to the end user
// (num cores can be set programmatically, so can gcpFile, etc.)
if (["-h", "--project-path",
"--zip-results", "--pmvs-num-cores", "--odm_georeferencing-useGcp",
"--start-with", "--odm_georeferencing-gcpFile", "--end-with"].indexOf(option) !== -1) continue;
let values = json[option];
let name = option.replace(/^--/, "");
let type = "";
let value = "";
let help = values.help || "";
let domain = values.metavar !== undefined ?
values.metavar.replace(/^[<>]/g, "")
.replace(/[<>]$/g, "")
.trim() :
"";
switch((values.type || "").trim()){
case "<type 'int'>":
type = "int";
value = values['default'] !== undefined ?
parseInt(values['default']) :
0;
break;
case "<type 'float'>":
type = "float";
value = values['default'] !== undefined ?
parseFloat(values['default']) :
0.0;
break;
default:
type = "string";
value = values['default'] !== undefined ?
values['default'].trim() :
"";
}
if (values['default'] === "True"){
type = "bool";
value = true;
}else if (values['default'] === "False"){
type = "bool";
value = false;
}
help = help.replace(/\%\(default\)s/g, value);
odmOptions.push({
name, type, value, domain, help
});
}
done(null, odmOptions);
}
});
},
// Checks that the options (as received from the rest endpoint)
// Are valid and within proper ranges.
// The result of filtering is passed back via callback
// @param options[]
filterOptions: function(options, done){
assert(odmOptions !== null, "odmOptions is not set. Have you initialized odmOptions properly?");
try{
if (typeof options === "string") options = JSON.parse(options);
let result = [];
let errors = [];
function addError(opt, descr){
errors.push({
name: opt.name,
error: descr
});
}
let typeConversion = {
'float': Number.parseFloat,
'int': Number.parseInt,
'bool': function(value){
if (value === 'true') return true;
else if (value === 'false') return false;
else if (typeof value === 'boolean') return value;
else throw new Error(`Cannot convert ${value} to boolean`);
}
};
let domainChecks = [
{
regex: /^(positive |negative )?(integer|float)$/,
validate: function(matches, value){
if (matches[1] === 'positive ') return value >= 0;
else if (matches[1] === 'negative ') return value <= 0;
else if (matches[2] === 'integer') return Number.isInteger(value);
else if (matches[2] === 'float') return Number.isFinite(value);
}
},
{
regex: /^percent$/,
validate: function(matches, value){
return value >= 0 && value <= 100;
}
},
{
regex: /^(float): ([\-\+\.\d]+) <= x <= ([\-\+\.\d]+)$/,
validate: function(matches, value){
let [str, type, lower, upper] = matches;
lower = parseFloat(lower);
upper = parseFloat(upper);
return value >= lower && value <= upper;
}
},
{
regex: /^(float) (>=|>|<|<=) ([\-\+\.\d]+)$/,
validate: function(matches, value){
let [str, type, oper, bound] = matches;
bound = parseFloat(bound);
switch(oper){
case '>=':
return value >= bound;
case '>':
return value > bound;
case '<=':
return value <= bound;
case '<':
return value < bound;
default:
return false;
}
}
}
];
function checkDomain(domain, value){
let dc, matches;
if (dc = domainChecks.find(dc => matches = domain.match(dc.regex))){
if (!dc.validate(matches, value)) throw new Error(`Invalid value ${value} (out of range)`);
}else{
throw new Error(`Domain value cannot be handled: '${domain}' : '${value}'`);
}
}
// Scan through all possible options
for (let odmOption of odmOptions){
// Was this option selected by the user?
let opt;
if (opt = options.find(o => o.name === odmOption.name)){
try{
// Convert to proper data type
let value = typeConversion[odmOption.type](opt.value);
// Domain check
if (odmOption.domain){
checkDomain(odmOption.domain, value);
}
result.push({
name: odmOption.name,
value: value
});
}catch(e){
addError(opt, e.message);
}
}
}
if (errors.length > 0) done(new Error(JSON.stringify(errors)));
else done(null, result);
}catch(e){
done(e);
}
}
};

64
libs/odmRunner.js
Wyświetl plik

@ -1,5 +1,5 @@
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
@ -17,32 +17,64 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
"use strict";
let spawn = require('child_process').spawn;
const ODM_PATH = "/code";
let config = require('../config.js');
let logger = require('./logger');
module.exports = {
run: function(options = {
projectPath: "/images"
"project-path": "/images"
}, done, outputReceived){
let command = [`${config.odm_path}/run.py`];
for (var name in options){
let value = options[name];
// Skip false booleans
if (value === false) continue;
command.push("--" + name);
// We don't specify "--time true" (just "--time")
if (typeof value !== 'boolean'){
command.push(value);
}
}
logger.info(`About to run: python ${command.join(" ")}`);
// Launch
let childProcess = spawn("python", [`${ODM_PATH}/run.py`,
"--project-path", options.projectPath
], {cwd: ODM_PATH});
let childProcess = spawn("python", command, {cwd: config.odm_path});
childProcess
.on('exit', (code, signal) => done(null, code, signal))
.on('error', done);
childProcess.stdout.on('data', chunk => outputReceived(chunk.toString()));
childProcess.stderr.on('data', chunk => outputReceived(chunk.toString()));
return childProcess;
},
getJsonOptions: function(done){
// Launch
let childProcess = spawn("python", [`${__dirname}/../helpers/odmOptionsToJson.py`,
"--project-path", config.odm_path]);
let output = [];
childProcess
.on('exit', (code, signal) => {
done(null, code, signal);
try{
let json = JSON.parse(output.join(""));
done(null, json);
}catch(err){
done(err);
}
})
.on('error', done);
childProcess.stdout.on('data', chunk => {
outputReceived(chunk.toString());
});
childProcess.stderr.on('data', chunk => {
outputReceived(chunk.toString());
});
let processOutput = chunk => output.push(chunk.toString());
return childProcess;
childProcess.stdout.on('data', processOutput);
childProcess.stderr.on('data', processOutput);
}
};

1
package.json
Wyświetl plik

@ -25,6 +25,7 @@
"body-parser": "^1.15.2",
"express": "^4.14.0",
"jsonfile": "^2.3.1",
"minimist": "^1.2.0",
"morgan": "^1.7.0",
"multer": "^1.1.0",
"node-schedule": "^1.1.1",

8
public/css/main.css
Wyświetl plik

@ -55,4 +55,12 @@
height: 200px;
font-family: monospace;
font-size: 90%;
}
.selectric-items li{
background: #fff;
}
#options .checkbox{
margin-right: 143px;
}

42
public/index.html
Wyświetl plik

@ -15,7 +15,6 @@
}
</style>
<link href="css/fileinput.css" media="all" rel="stylesheet" type="text/css" />
<link rel="stylesheet" href="css/main.css">
<script src="js/vendor/modernizr-2.8.3.min.js"></script>
@ -41,20 +40,42 @@
<br/>
<form enctype="multipart/form-data" onsubmit="return false;">
<div class="form-group form-inline">
<label for="taskName">Project Name:</lable> <input type="text" class="form-control" value="" id="taskName"/>
<label for="taskName">Project Name:</lable> <input type="text" class="form-control" value="" id="taskName" />
</div>
<div class="form-group">
<label for="images">Aerial Imageries:</label> <input id="images" name="images" multiple type="file">
<div id="errorBlock" class="help-block"></div>
</div>
<div class="form-group">
<!-- <label>Options:</label> -->
<label for="images">Aerial Imageries and GCP List (optional):</label> <input id="images" name="images" multiple type="file">
<div id="errorBlock" class="help-block"></div>
</div>
<div class="text-right"><input type="submit" class="btn btn-success" value="Start Task" id="btnUpload" /></div>
<div id="options">
<div class="form-inline form-group form-horizontal">
<div data-bind="visible: error(), text: error()" class="alert alert-warning" role="alert"></div>
<button style="position: relative; top: -45px;" type="submit" class="btn btn-default" data-bind="visible: !error(), click: function(){ showOptions(!showOptions()); }, text: (showOptions() ? 'Hide' : 'Show') + ' Options'"></button>
<div data-bind="visible: showOptions()">
<div data-bind="foreach: options">
<label data-bind="text: properties.name + (properties.domain ? ' (' + properties.domain + ')' : '')"></label><br/>
<!-- ko if: properties.type !== 'bool' -->
<input type="text" class="form-control" data-bind="attr: {placeholder: properties.value}, value: value">
<!-- /ko -->
<!-- ko if: properties.type === 'bool' -->
<div class="checkbox">
<label>
<input type="checkbox" data-bind="checked: value"> Enable
</label>
</div>
<!-- /ko -->
<button type="submit" class="btn glyphicon glyphicon-info-sign btn-info" data-toggle="tooltip" data-placement="top" data-bind="attr: {title: properties.help}"></button>
<button type="submit" class="btn glyphicon glyphicon glyphicon-repeat btn-default" data-toggle="tooltip" data-placement="top" title="Reset to default" data-bind="click: resetToDefault"></button>
<div class="text-right"><input type="submit" class="btn btn-success" value="Start Task" id="btnUpload" /></div>
</form>
<br/><br/>
</div>
</div>
</div>
</div>
</form>
</div>
<div class="col-md-7">
<div class="col-md-7" id="taskList">
<h2>Current Tasks (<span data-bind="text: tasks().length"></span>)</h2>
<p data-bind="visible: tasks().length === 0">No running tasks.</p>
<div data-bind="foreach: tasks">
@ -113,6 +134,7 @@
<script src="js/vendor/bootstrap.min.js"></script>
<script src="js/vendor/knockout-3.4.0.js"></script>
<script src="js/fileinput.js" type="text/javascript"></script>
<script src="js/main.js"></script>
</body>
</html>

4
public/js/fileinput.js
Wyświetl plik

@ -3205,8 +3205,8 @@
$.fn.fileinputLocales.en = {
fileSingle: 'file',
filePlural: 'files',
browseLabel: 'Browse &hellip;',
removeLabel: 'Remove',
browseLabel: 'Add &hellip;',
removeLabel: 'Clear All',
removeTitle: 'Clear selected files',
cancelLabel: 'Cancel',
cancelTitle: 'Abort ongoing upload',

4
public/js/locales/LANG.js
Wyświetl plik

@ -14,8 +14,8 @@
$.fn.fileinputLocales['_LANG_'] = {
fileSingle: 'file',
filePlural: 'files',
browseLabel: 'Browse &hellip;',
removeLabel: 'Remove',
browseLabel: 'Add &hellip;',
removeLabel: 'Clear All',
removeTitle: 'Clear selected files',
cancelLabel: 'Cancel',
cancelTitle: 'Abort ongoing upload',

121
public/js/main.js
Wyświetl plik

@ -59,6 +59,14 @@ $(function(){
this.saveTaskListToLocalStorage();
};
var codes = {
QUEUED: 10,
RUNNING: 20,
FAILED: 30,
COMPLETED: 40,
CANCELED: 50
};
function Task(uuid){
var self = this;
@ -70,13 +78,6 @@ $(function(){
this.resetOutput();
this.timeElapsed = ko.observable("00:00:00");
var codes = {
QUEUED: 10,
RUNNING: 20,
FAILED: 30,
COMPLETED: 40,
CANCELED: 50
};
var statusCodes = {
10: {
descr: "Queued",
@ -149,8 +150,8 @@ $(function(){
})
.always(function(){ self.loading(false); });
};
Task.prototype.consoleMouseOver = function(){ this.autoScrollOutput = false; }
Task.prototype.consoleMouseOut = function(){ this.autoScrollOutput = true; }
Task.prototype.consoleMouseOver = function(){ this.autoScrollOutput = false; };
Task.prototype.consoleMouseOut = function(){ this.autoScrollOutput = true; };
Task.prototype.resetOutput = function(){
this.viewOutputLine = 0;
this.autoScrollOutput = true;
@ -170,7 +171,7 @@ $(function(){
self.viewOutputLine += output.length;
if (self.autoScrollOutput){
var $console = $("#console_" + self.uuid);
$console.scrollTop($console[0].scrollHeight - $console.height())
$console.scrollTop($console[0].scrollHeight - $console.height());
}
}
})
@ -205,22 +206,30 @@ $(function(){
var self = this;
var url = "/task/remove";
$.post(url, {
uuid: this.uuid
})
.done(function(json){
if (json.success || self.info().error){
taskList.remove(self);
}else{
self.info({error: json.error});
}
function doRemove(){
$.post(url, {
uuid: self.uuid
})
.done(function(json){
if (json.success || self.info().error){
taskList.remove(self);
}else{
self.info({error: json.error});
}
self.stopRefreshingInfo();
})
.fail(function(){
self.info({error: url + " is unreachable."});
self.stopRefreshingInfo();
});
self.stopRefreshingInfo();
})
.fail(function(){
self.info({error: url + " is unreachable."});
self.stopRefreshingInfo();
});
}
if (this.info().status && this.info().status.code === codes.COMPLETED){
if (confirm("Are you sure?")) doRemove();
}else{
doRemove();
}
};
function genApiCall(url, onSuccess){
@ -243,8 +252,8 @@ $(function(){
self.info({error: url + " is unreachable."});
self.stopRefreshingInfo();
});
}
};
};
}
Task.prototype.cancel = genApiCall("/task/cancel");
Task.prototype.restart = genApiCall("/task/restart", function(task){
task.resetOutput();
@ -254,19 +263,20 @@ $(function(){
};
var taskList = new TaskList();
ko.applyBindings(taskList);
ko.applyBindings(taskList, document.getElementById('taskList'));
// Handle uploads
$("#images").fileinput({
uploadUrl: '/task/new',
showPreview: false,
allowedFileExtensions: ['jpg', 'jpeg'],
allowedFileExtensions: ['jpg', 'jpeg', 'txt'],
elErrorContainer: '#errorBlock',
showUpload: false,
uploadAsync: false,
uploadExtraData: function(){
return {
name: $("#taskName").val()
name: $("#taskName").val(),
options: JSON.stringify(optionsModel.getUserOptions())
};
}
});
@ -294,4 +304,55 @@ $(function(){
})
.on('filebatchuploaderror', function(e, data, msg){
});
// Load options
function Option(properties){
this.properties = properties;
this.value = ko.observable();
}
Option.prototype.resetToDefault = function(){
this.value(undefined);
};
function OptionsModel(){
var self = this;
this.options = ko.observableArray();
this.options.subscribe(function(){
setTimeout(function(){
$('#options [data-toggle="tooltip"]').tooltip();
}, 100);
});
this.showOptions = ko.observable(false);
this.error = ko.observable();
$.get("/getOptions")
.done(function(json){
if (json.error) self.error(json.error);
else{
for (var i in json){
self.options.push(new Option(json[i]));
}
}
})
.fail(function(){
self.error("options are not available.");
});
}
OptionsModel.prototype.getUserOptions = function(){
var result = [];
for (var i = 0; i < this.options().length; i++){
var opt = this.options()[i];
if (opt.value() !== undefined){
result.push({
name: opt.properties.name,
value: opt.value()
});
}
}
return result;
};
var optionsModel = new OptionsModel();
ko.applyBindings(optionsModel, document.getElementById("options"));
});

BIN
screenshots/main.png
Wyświetl plik

Plik binarny nie jest wyświetlany.
Side by Side Swipe Nakładka

Przed

Szerokość:  |  Wysokość:  |  Rozmiar: 72 KiB

Po

Szerokość:  |  Wysokość:  |  Rozmiar: 100 KiB