mirror
/
OpenDroneMap-NodeODM
kopia lustrzana https://github.com/OpenDroneMap/NodeODM
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność

Merge pull request #70 from pierotofy/chunked 

Chunked
pull/73/head
Piero Toffanin 2019-02-04 09:55:13 -05:00 zatwierdzone przez GitHub
rodzic 911bf08b1b 06fef5549b
commit 4463ae6ec9
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
17 zmienionych plików z 4657 dodań i 3591 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

7
README.md
Wyświetl plik

@ -160,9 +160,4 @@ Make a pull request for small contributions. For big contributions, please open
## Roadmap
- [X] Command line options for OpenDroneMap
- [X] GPC List support
- [ ] Video support when the [SLAM module](https://github.com/OpenDroneMap/OpenDroneMap/pull/317) becomes available
- [ ] Continuous Integration Setup
- [X] Documentation
- [ ] Unit Testing
See the [list of wanted features](https://github.com/OpenDroneMap/NodeODM/issues?q=is%3Aopen+is%3Aissue+label%3A%22new+feature%22).

4
config.js
Wyświetl plik

@ -33,9 +33,11 @@ Options:
-d, --deamonize Set process to run as a deamon
--parallel_queue_processing <number> Number of simultaneous processing tasks (default: 2)
--cleanup_tasks_after <number> Number of minutes that elapse before deleting finished and canceled tasks (default: 2880)
--cleanup_uploads_after <number> Number of minutes that elapse before deleting unfinished uploads. Set this value to the maximum time you expect a dataset to be uploaded. (default: 2880)
--test Enable test mode. In test mode, no commands are sent to OpenDroneMap. This can be useful during development or testing (default: false)
--test_skip_orthophotos If test mode is enabled, skip orthophoto results when generating assets. (default: false)
--test_skip_dems If test mode is enabled, skip dems results when generating assets. (default: false)
--test_drop_uploads If test mode is enabled, drop /task/new/upload requests with 50% probability. (default: false)
--powercycle When set, the application exits immediately after powering up. Useful for testing launch and compilation issues.
--token <token> Sets a token that needs to be passed for every request. This can be used to limit access to the node only to token holders. (default: none)
--max_images <number> Specify the maximum number of images that this processing node supports. (default: unlimited)
@ -90,9 +92,11 @@ config.port = parseInt(argv.port || argv.p || fromConfigFile("port", process.env
config.deamon = argv.deamonize || argv.d || fromConfigFile("daemon", false);
config.parallelQueueProcessing = argv.parallel_queue_processing || fromConfigFile("parallelQueueProcessing", 2);
config.cleanupTasksAfter = parseInt(argv.cleanup_tasks_after || fromConfigFile("cleanupTasksAfter", 2880));
config.cleanupUploadsAfter = parseInt(argv.cleanup_uploads_after || fromConfigFile("cleanupUploadsAfter", 2880));
config.test = argv.test || fromConfigFile("test", false);
config.testSkipOrthophotos = argv.test_skip_orthophotos || fromConfigFile("testSkipOrthophotos", false);
config.testSkipDems = argv.test_skip_dems || fromConfigFile("testSkipDems", false);
config.testDropUploads = argv.test_drop_uploads || fromConfigFile("testDropUploads", false);
config.powercycle = argv.powercycle || fromConfigFile("powercycle", false);
config.token = argv.token || fromConfigFile("token", "");
config.maxImages = parseInt(argv.max_images || fromConfigFile("maxImages", "")) || null;

143
docs/index.adoc
Wyświetl plik

@ -8,7 +8,7 @@ REST API to access ODM
=== Version information
[%hardbreaks]
_Version_ : 1.3.1
_Version_ : 1.4.0
=== Contact information
@ -281,7 +281,7 @@ _required_|UUID of the task|string|
=== POST /task/new
==== Description
Creates a new task and places it at the end of the processing queue
Creates a new task and places it at the end of the processing queue. For uploading really large tasks, see /task/new/init instead.
==== Parameters
@ -301,6 +301,8 @@ _optional_|An optional name to be associated with the task|string|
_optional_|Serialized JSON string of the options to use for processing, as an array of the format: [{name: option1, value: value1}, {name: option2, value: value2}, …]. For example, [{"name":"cmvs-maxImages","value":"500"},{"name":"time","value":true}]. For a list of all options, call /options|string|
|*FormData*|*skipPostProcessing* +
_optional_|When set, skips generation of map tiles, derivate assets, point cloud tiles.|boolean|
|*FormData*|*webhook* +
_optional_|Optional URL to call when processing has ended (either successfully or unsuccessfully).|string|
|*FormData*|*zipurl* +
_optional_|URL of the zip file containing the images to process, plus an optional GCP file. If included, the GCP file should have .txt extension|string|
|===
@ -336,6 +338,143 @@ _required_|UUID of the newly created task|string
* task
[[_task_new_commit_uuid_post]]
=== POST /task/new/commit/{uuid}
==== Description
Creates a new task for which images have been uploaded via /task/new/upload.
==== Parameters
[options="header", cols=".^2,.^3,.^9,.^4,.^2"]
|===
|Type|Name|Description|Schema|Default
|*Path*|*uuid* +
_required_|UUID of the task|string|
|*Query*|*token* +
_optional_|Token required for authentication (when authentication is required).|string|
|===
==== Responses
[options="header", cols=".^2,.^14,.^4"]
|===
|HTTP Code|Description|Schema
|*200*|Success|<<_task_new_commit_uuid_post_response_200,Response 200>>
|*default*|Error|<<_error,Error>>
|===
[[_task_new_commit_uuid_post_response_200]]
*Response 200*
[options="header", cols=".^3,.^11,.^4"]
|===
|Name|Description|Schema
|*uuid* +
_required_|UUID of the newly created task|string
|===
==== Tags
* task
[[_task_new_init_post]]
=== POST /task/new/init
==== Description
Initialize the upload of a new task. If successful, a user can start uploading files via /task/new/upload. The task will not start until /task/new/commit is called.
==== Parameters
[options="header", cols=".^2,.^3,.^9,.^4,.^2"]
|===
|Type|Name|Description|Schema|Default
|*Header*|*set-uuid* +
_optional_|An optional UUID string that will be used as UUID for this task instead of generating a random one.|string|
|*Query*|*token* +
_optional_|Token required for authentication (when authentication is required).|string|
|*FormData*|*name* +
_optional_|An optional name to be associated with the task|string|
|*FormData*|*options* +
_optional_|Serialized JSON string of the options to use for processing, as an array of the format: [{name: option1, value: value1}, {name: option2, value: value2}, …]. For example, [{"name":"cmvs-maxImages","value":"500"},{"name":"time","value":true}]. For a list of all options, call /options|string|
|*FormData*|*skipPostProcessing* +
_optional_|When set, skips generation of map tiles, derivate assets, point cloud tiles.|boolean|
|*FormData*|*webhook* +
_optional_|Optional URL to call when processing has ended (either successfully or unsuccessfully).|string|
|===
==== Responses
[options="header", cols=".^2,.^14,.^4"]
|===
|HTTP Code|Description|Schema
|*200*|Success|<<_task_new_init_post_response_200,Response 200>>
|*default*|Error|<<_error,Error>>
|===
[[_task_new_init_post_response_200]]
*Response 200*
[options="header", cols=".^3,.^11,.^4"]
|===
|Name|Description|Schema
|*uuid* +
_required_|UUID of the newly created task|string
|===
==== Tags
* task
[[_task_new_upload_uuid_post]]
=== POST /task/new/upload/{uuid}
==== Description
Adds one or more files to the task created via /task/new/init. It does not start the task. To start the task, call /task/new/commit.
==== Parameters
[options="header", cols=".^2,.^3,.^9,.^4,.^2"]
|===
|Type|Name|Description|Schema|Default
|*Path*|*uuid* +
_required_|UUID of the task|string|
|*Query*|*token* +
_optional_|Token required for authentication (when authentication is required).|string|
|*FormData*|*images* +
_required_|Images to process, plus an optional GCP file. If included, the GCP file should have .txt extension|file|
|===
==== Responses
[options="header", cols=".^2,.^14,.^4"]
|===
|HTTP Code|Description|Schema
|*200*|File Received|<<_response,Response>>
|*default*|Error|<<_error,Error>>
|===
==== Consumes
* `multipart/form-data`
==== Tags
* task
[[_task_remove_post]]
=== POST /task/remove

2
docs/swagger.json
Wyświetl plik

File diff suppressed because one or more lines are too long

353
index.js
Wyświetl plik

@ -22,74 +22,169 @@ const config = require('./config.js');
const packageJson = JSON.parse(fs.readFileSync('./package.json'));
const logger = require('./libs/logger');
const path = require('path');
const async = require('async');
const mime = require('mime');
const rmdir = require('rimraf');
const express = require('express');
const app = express();
const multer = require('multer');
const bodyParser = require('body-parser');
const multer = require('multer');
const TaskManager = require('./libs/TaskManager');
const Task = require('./libs/Task');
const odmInfo = require('./libs/odmInfo');
const Directories = require('./libs/Directories');
const unzip = require('node-unzip-2');
const si = require('systeminformation');
const mv = require('mv');
const S3 = require('./libs/S3');
const auth = require('./libs/auth/factory').fromConfig(config);
const authCheck = auth.getMiddleware();
const uuidv4 = require('uuid/v4');
// zip files
let request = require('request');
let download = function(uri, filename, callback) {
request.head(uri, function(err, res, body) {
if (err) callback(err);
else{
request(uri).pipe(fs.createWriteStream(filename)).on('close', callback);
}
});
};
const taskNew = require('./libs/taskNew');
app.use(express.static('public'));
app.use('/swagger.json', express.static('docs/swagger.json'));
const upload = multer({
storage: multer.diskStorage({
destination: (req, file, cb) => {
let dstPath = path.join("tmp", req.id);
fs.exists(dstPath, exists => {
if (!exists) {
fs.mkdir(dstPath, undefined, () => {
cb(null, dstPath);
});
} else {
cb(null, dstPath);
}
});
},
filename: (req, file, cb) => {
cb(null, file.originalname);
}
})
});
const formDataParser = multer().none();
const urlEncodedBodyParser = bodyParser.urlencoded({extended: false});
let taskManager;
let server;
/** @swagger
* /task/new/init:
* post:
* description: Initialize the upload of a new task. If successful, a user can start uploading files via /task/new/upload. The task will not start until /task/new/commit is called.
* tags: [task]
* parameters:
* -
* name: name
* in: formData
* description: An optional name to be associated with the task
* required: false
* type: string
* -
* name: options
* in: formData
* description: 'Serialized JSON string of the options to use for processing, as an array of the format: [{name: option1, value: value1}, {name: option2, value: value2}, ...]. For example, [{"name":"cmvs-maxImages","value":"500"},{"name":"time","value":true}]. For a list of all options, call /options'
* required: false
* type: string
* -
* name: skipPostProcessing
* in: formData
* description: 'When set, skips generation of map tiles, derivate assets, point cloud tiles.'
* required: false
* type: boolean
* -
* name: webhook
* in: formData
* description: Optional URL to call when processing has ended (either successfully or unsuccessfully).
* required: false
* type: string
* -
* name: token
* in: query
* description: 'Token required for authentication (when authentication is required).'
* required: false
* type: string
* -
* name: set-uuid
* in: header
* description: 'An optional UUID string that will be used as UUID for this task instead of generating a random one.'
* required: false
* type: string
* responses:
* 200:
* description: Success
* schema:
* type: object
* required: [uuid]
* properties:
* uuid:
* type: string
* description: UUID of the newly created task
* default:
* description: Error
* schema:
* $ref: '#/definitions/Error'
*/
app.post('/task/new/init', authCheck, taskNew.assignUUID, formDataParser, taskNew.handleInit);
/** @swagger
* /task/new/upload/{uuid}:
* post:
* description: Adds one or more files to the task created via /task/new/init. It does not start the task. To start the task, call /task/new/commit.
* tags: [task]
* consumes:
* - multipart/form-data
* parameters:
* -
* name: uuid
* in: path
* description: UUID of the task
* required: true
* type: string
* -
* name: images
* in: formData
* description: Images to process, plus an optional GCP file. If included, the GCP file should have .txt extension
* required: true
* type: file
* -
* name: token
* in: query
* description: 'Token required for authentication (when authentication is required).'
* required: false
* type: string
* responses:
* 200:
* description: File Received
* schema:
* $ref: "#/definitions/Response"
* default:
* description: Error
* schema:
* $ref: '#/definitions/Error'
*/
app.post('/task/new/upload/:uuid', authCheck, taskNew.getUUID, taskNew.preUpload, taskNew.uploadImages, taskNew.handleUpload);
/** @swagger
* /task/new/commit/{uuid}:
* post:
* description: Creates a new task for which images have been uploaded via /task/new/upload.
* tags: [task]
* parameters:
* -
* name: uuid
* in: path
* description: UUID of the task
* required: true
* type: string
* -
* name: token
* in: query
* description: 'Token required for authentication (when authentication is required).'
* required: false
* type: string
* responses:
* 200:
* description: Success
* schema:
* type: object
* required: [uuid]
* properties:
* uuid:
* type: string
* description: UUID of the newly created task
* default:
* description: Error
* schema:
* $ref: '#/definitions/Error'
*/
app.post('/task/new/commit/:uuid', authCheck, taskNew.getUUID, taskNew.handleCommit, taskNew.createTask);
/** @swagger
* /task/new:
* post:
* description: Creates a new task and places it at the end of the processing queue
* description: Creates a new task and places it at the end of the processing queue. For uploading really large tasks, see /task/new/init instead.
* tags: [task]
* consumes:
* - multipart/form-data
@ -125,6 +220,12 @@ let server;
* required: false
* type: boolean
* -
* name: webhook
* in: formData
* description: Optional URL to call when processing has ended (either successfully or unsuccessfully).
* required: false
* type: string
* -
* name: token
* in: query
* description: 'Token required for authentication (when authentication is required).'
@ -151,158 +252,12 @@ let server;
* schema:
* $ref: '#/definitions/Error'
*/
app.post('/task/new', authCheck, (req, res, next) => {
// A user can optionally suggest a UUID instead of letting
// nodeODM pick one.
if (req.get('set-uuid')){
const userUuid = req.get('set-uuid');
// Valid UUID and no other task with same UUID?
if (/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(userUuid) && !taskManager.find(userUuid)){
req.id = userUuid;
next();
}else{
res.json({error: `Invalid set-uuid: ${userUuid}`})
}
}else{
req.id = uuidv4();
next();
}
}, upload.array('images'), (req, res) => {
// TODO: consider doing the file moving in the background
// and return a response more quickly instead of a long timeout.
req.setTimeout(1000 * 60 * 20);
let srcPath = path.join("tmp", req.id);
// Print error message and cleanup
const die = (error) => {
res.json({error});
// Check if tmp/ directory needs to be cleaned
if (fs.stat(srcPath, (err, stats) => {
if (!err && stats.isDirectory()) rmdir(srcPath, () => {}); // ignore errors, don't wait
}));
};
if ((!req.files || req.files.length === 0) && !req.body.zipurl) die("Need at least 1 file or a zip file url.");
else if (config.maxImages && req.files && req.files.length > config.maxImages) die(`${req.files.length} images uploaded, but this node can only process up to ${config.maxImages}.`);
else {
let destPath = path.join(Directories.data, req.id);
let destImagesPath = path.join(destPath, "images");
let destGpcPath = path.join(destPath, "gpc");
async.series([
cb => {
odmInfo.filterOptions(req.body.options, (err, options) => {
if (err) cb(err);
else {
req.body.options = options;
cb(null);
}
});
},
// Move all uploads to data/<uuid>/images dir (if any)
cb => {
if (req.files && req.files.length > 0) {
fs.stat(destPath, (err, stat) => {
if (err && err.code === 'ENOENT') cb();
else cb(new Error(`Directory exists (should not have happened: ${err.code})`));
});
} else {
cb();
}
},
// Unzips zip URL to tmp/<uuid>/ (if any)
cb => {
if (req.body.zipurl) {
let archive = "zipurl.zip";
upload.storage.getDestination(req, archive, (err, dstPath) => {
if (err) cb(err);
else{
let archiveDestPath = path.join(dstPath, archive);
download(req.body.zipurl, archiveDestPath, cb);
}
});
} else {
cb();
}
},
cb => fs.mkdir(destPath, undefined, cb),
cb => fs.mkdir(destGpcPath, undefined, cb),
cb => mv(srcPath, destImagesPath, cb),
cb => {
// Find any *.zip file and extract
fs.readdir(destImagesPath, (err, entries) => {
if (err) cb(err);
else {
async.eachSeries(entries, (entry, cb) => {
if (/\.zip$/gi.test(entry)) {
let filesCount = 0;
fs.createReadStream(path.join(destImagesPath, entry)).pipe(unzip.Parse())
.on('entry', function(entry) {
if (entry.type === 'File') {
filesCount++;
entry.pipe(fs.createWriteStream(path.join(destImagesPath, path.basename(entry.path))));
} else {
entry.autodrain();
}
})
.on('close', () => {
// Verify max images limit
if (config.maxImages && filesCount > config.maxImages) cb(`${filesCount} images uploaded, but this node can only process up to ${config.maxImages}.`);
else cb();
})
.on('error', cb);
} else cb();
}, cb);
}
});
},
cb => {
// Find any *.txt (GPC) file and move it to the data/<uuid>/gpc directory
// also remove any lingering zipurl.zip
fs.readdir(destImagesPath, (err, entries) => {
if (err) cb(err);
else {
async.eachSeries(entries, (entry, cb) => {
if (/\.txt$/gi.test(entry)) {
mv(path.join(destImagesPath, entry), path.join(destGpcPath, entry), cb);
}else if (/\.zip$/gi.test(entry)){
fs.unlink(path.join(destImagesPath, entry), cb);
} else cb();
}, cb);
}
});
},
// Create task
cb => {
new Task(req.id, req.body.name, (err, task) => {
if (err) cb(err);
else {
taskManager.addNew(task);
res.json({ uuid: req.id });
cb();
}
}, req.body.options,
req.body.webhook,
req.body.skipPostProcessing === 'true');
}
], err => {
if (err) die(err.message);
});
}
});
app.post('/task/new', authCheck, taskNew.assignUUID, taskNew.uploadImages, (req, res, next) => {
req.body = req.body || {};
if ((!req.files || req.files.length === 0) && !req.body.zipurl) req.error = "Need at least 1 file or a zip file url.";
else if (config.maxImages && req.files && req.files.length > config.maxImages) req.error = `${req.files.length} images uploaded, but this node can only process up to ${config.maxImages}.`;
next();
}, taskNew.createTask);
let getTaskFromUuid = (req, res, next) => {
let task = taskManager.find(req.params.uuid);
@ -857,13 +812,21 @@ process.on('SIGTERM', gracefulShutdown);
process.on('SIGINT', gracefulShutdown);
// Startup
if (config.test) logger.info("Running in test mode");
if (config.test) {
logger.info("Running in test mode");
if (config.testSkipOrthophotos) logger.info("Orthophotos will be skipped");
if (config.testSkipDems) logger.info("DEMs will be skipped");
if (config.testDropUploads) logger.info("Uploads will drop at random");
}
let commands = [
cb => odmInfo.initialize(cb),
cb => auth.initialize(cb),
cb => S3.initialize(cb),
cb => { taskManager = new TaskManager(cb); },
cb => {
TaskManager.initialize(cb);
taskManager = TaskManager.singleton();
},
cb => {
server = app.listen(config.port, err => {
if (!err) logger.info('Server has started on port ' + String(config.port));

18
libs/Task.js
Wyświetl plik

@ -46,7 +46,7 @@ module.exports = class Task{
this.processingTime = -1;
this.setStatus(statusCodes.QUEUED);
this.options = options;
this.gpcFiles = [];
this.gcpFiles = [];
this.output = [];
this.runningProcesses = [];
this.webhook = webhook;
@ -67,15 +67,15 @@ module.exports = class Task{
// Find GCP (if any)
cb => {
fs.readdir(this.getGpcFolderPath(), (err, files) => {
fs.readdir(this.getGcpFolderPath(), (err, files) => {
if (err) cb(err);
else{
files.forEach(file => {
if (/\.txt$/gi.test(file)){
this.gpcFiles.push(file);
this.gcpFiles.push(file);
}
});
logger.debug(`Found ${this.gpcFiles.length} GPC files (${this.gpcFiles.join(" ")}) for ${this.uuid}`);
logger.debug(`Found ${this.gcpFiles.length} GCP files (${this.gcpFiles.join(" ")}) for ${this.uuid}`);
cb(null);
}
});
@ -110,10 +110,10 @@ module.exports = class Task{
return path.join(this.getProjectFolderPath(), "images");
}
// Get path where GPC file(s) are stored
// Get path where GCP file(s) are stored
// (relative to nodejs process CWD)
getGpcFolderPath(){
return path.join(this.getProjectFolderPath(), "gpc");
getGcpFolderPath(){
return path.join(this.getProjectFolderPath(), "gcp");
}
// Get path of project (where all images and assets folder are contained)
@ -385,8 +385,8 @@ module.exports = class Task{
runnerOptions["project-path"] = fs.realpathSync(Directories.data);
if (this.gpcFiles.length > 0){
runnerOptions.gcp = fs.realpathSync(path.join(this.getGpcFolderPath(), this.gpcFiles[0]));
if (this.gcpFiles.length > 0){
runnerOptions.gcp = fs.realpathSync(path.join(this.getGcpFolderPath(), this.gcpFiles[0]));
}
this.runningProcesses.push(odmRunner.run(runnerOptions, this.uuid, (err, code, signal) => {

39
libs/TaskManager.js
Wyświetl plik

@ -30,8 +30,11 @@ const Directories = require('./Directories');
const TASKS_DUMP_FILE = path.join(Directories.data, "tasks.json");
const CLEANUP_TASKS_IF_OLDER_THAN = 1000 * 60 * config.cleanupTasksAfter; // minutes
const CLEANUP_STALE_UPLOADS_AFTER = 1000 * 60 * config.cleanupUploadsAfter; // minutes
module.exports = class TaskManager{
let taskManager;
class TaskManager{
constructor(done){
this.tasks = {};
this.runningQueue = [];
@ -40,6 +43,7 @@ module.exports = class TaskManager{
cb => this.restoreTaskListFromDump(cb),
cb => this.removeOldTasks(cb),
cb => this.removeOrphanedDirectories(cb),
cb => this.removeStaleUploads(cb),
cb => {
this.processNextTask();
cb();
@ -49,6 +53,7 @@ module.exports = class TaskManager{
schedule.scheduleJob('0 * * * *', () => {
this.removeOldTasks();
this.dumpTaskList();
this.removeStaleUploads();
});
cb();
@ -101,6 +106,29 @@ module.exports = class TaskManager{
});
}
removeStaleUploads(done){
fs.readdir("tmp", (err, entries) => {
if (err) done(err);
else{
const now = new Date();
async.eachSeries(entries, (entry, cb) => {
let dirPath = path.join("tmp", entry);
if (entry.match(/^[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+$/)){
fs.stat(dirPath, (err, stats) => {
if (err) cb(err);
else{
if (stats.isDirectory() && stats.ctime.getTime() + CLEANUP_STALE_UPLOADS_AFTER < now.getTime()){
logger.info(`Found stale upload directory: ${entry}, removing...`);
rmdir(dirPath, cb);
}else cb();
}
});
}else cb();
}, done);
}
});
}
// Load tasks that already exists (if any)
restoreTaskListFromDump(done){
fs.readFile(TASKS_DUMP_FILE, (err, data) => {
@ -264,4 +292,11 @@ module.exports = class TaskManager{
}
return count;
}
};
}
module.exports = {
singleton: function(){ return taskManager; },
initialize: function(cb){
taskManager = new TaskManager(cb);
}
};

1
libs/odmInfo.js
Wyświetl plik

@ -123,6 +123,7 @@ module.exports = {
if (domain.indexOf(value) === -1) domain.unshift(value);
}
help = help.replace(/^One of: \%\(choices\)s. /, "");
help = help.replace(/\%\(default\)s/g, value);
odmOptions.push({

337
libs/taskNew.js 100644
Wyświetl plik

@ -0,0 +1,337 @@
/*
Node-OpenDroneMap Node.js App and REST API to access OpenDroneMap.
Copyright (C) 2016 Node-OpenDroneMap Contributors
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
const multer = require('multer');
const fs = require('fs');
const path = require('path');
const TaskManager = require('./TaskManager');
const uuidv4 = require('uuid/v4');
const config = require('../config.js');
const rmdir = require('rimraf');
const Directories = require('./Directories');
const unzip = require('node-unzip-2');
const mv = require('mv');
const Task = require('./Task');
const async = require('async');
const odmInfo = require('./odmInfo');
const request = require('request');
const utils = require('./utils');
const download = function(uri, filename, callback) {
request.head(uri, function(err, res, body) {
if (err) callback(err);
else{
request(uri).pipe(fs.createWriteStream(filename)).on('close', callback);
}
});
};
const removeDirectory = function(dir, cb = () => {}){
fs.stat(dir, (err, stats) => {
if (!err && stats.isDirectory()) rmdir(dir, cb); // ignore errors, don't wait
else cb(err);
});
};
const upload = multer({
storage: multer.diskStorage({
destination: (req, file, cb) => {
let dstPath = path.join("tmp", req.id);
fs.exists(dstPath, exists => {
if (!exists) {
fs.mkdir(dstPath, undefined, () => {
cb(null, dstPath);
});
} else {
cb(null, dstPath);
}
});
},
filename: (req, file, cb) => {
let filename = utils.sanitize(file.originalname);
if (filename === "body.json") filename = "_body.json";
cb(null, filename);
}
})
});
module.exports = {
assignUUID: (req, res, next) => {
// A user can optionally suggest a UUID instead of letting
// nodeODM pick one.
if (req.get('set-uuid')){
const userUuid = req.get('set-uuid');
// Valid UUID and no other task with same UUID?
if (/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(userUuid) && !TaskManager.singleton().find(userUuid)){
req.id = userUuid;
next();
}else{
res.json({error: `Invalid set-uuid: ${userUuid}`})
}
}else{
req.id = uuidv4();
next();
}
},
getUUID: (req, res, next) => {
req.id = req.params.uuid;
if (!req.id) res.json({error: `Invalid uuid (not set)`});
const srcPath = path.join("tmp", req.id);
const bodyFile = path.join(srcPath, "body.json");
fs.access(bodyFile, fs.F_OK, err => {
if (err) res.json({error: `Invalid uuid (not found)`});
else next();
});
},
preUpload: (req, res, next) => {
// Testing stuff
if (!config.test) next();
else{
if (config.testDropUploads){
if (Math.random() < 0.5) res.sendStatus(500);
else next();
}else{
next();
}
}
},
uploadImages: upload.array("images"),
handleUpload: (req, res) => {
// IMPROVEMENT: check files count limits ahead of handleTaskNew
if (req.files && req.files.length > 0){
res.json({success: true});
}else{
res.json({error: "Need at least 1 file."});
}
},
handleCommit: (req, res, next) => {
const srcPath = path.join("tmp", req.id);
const bodyFile = path.join(srcPath, "body.json");
async.series([
cb => {
fs.readFile(bodyFile, 'utf8', (err, data) => {
if (err) cb(err);
else{
try{
const body = JSON.parse(data);
fs.unlink(bodyFile, err => {
if (err) cb(err);
else cb(null, body);
});
}catch(e){
cb("Malformed body.json");
}
}
});
},
cb => fs.readdir(srcPath, cb),
], (err, [ body, files ]) => {
if (err) res.json({error: err.message});
else{
req.body = body;
req.files = files;
if (req.files.length === 0){
req.error = "Need at least 1 file.";
}
next();
}
});
},
handleInit: (req, res) => {
req.body = req.body || {};
const srcPath = path.join("tmp", req.id);
const bodyFile = path.join(srcPath, "body.json");
// Print error message and cleanup
const die = (error) => {
res.json({error});
removeDirectory(srcPath);
};
async.series([
cb => {
// Check for problems before file uploads
if (req.body && req.body.options){
odmInfo.filterOptions(req.body.options, err => {
if (err) cb(err);
else cb();
});
}else cb();
},
cb => {
fs.stat(srcPath, (err, stat) => {
if (err && err.code === 'ENOENT') cb();
else cb(new Error(`Directory exists (should not have happened: ${err.code})`));
});
},
cb => fs.mkdir(srcPath, undefined, cb),
cb => {
fs.writeFile(bodyFile, JSON.stringify(req.body), {encoding: 'utf8'}, cb);
},
cb => {
res.json({uuid: req.id});
cb();
}
], err => {
if (err) die(err.message);
});
},
createTask: (req, res) => {
// IMPROVEMENT: consider doing the file moving in the background
// and return a response more quickly instead of a long timeout.
req.setTimeout(1000 * 60 * 20);
const srcPath = path.join("tmp", req.id);
// Print error message and cleanup
const die = (error) => {
res.json({error});
removeDirectory(srcPath);
};
if (req.error !== undefined){
die(req.error);
}else{
let destPath = path.join(Directories.data, req.id);
let destImagesPath = path.join(destPath, "images");
let destGcpPath = path.join(destPath, "gcp");
async.series([
cb => {
odmInfo.filterOptions(req.body.options, (err, options) => {
if (err) cb(err);
else {
req.body.options = options;
cb(null);
}
});
},
// Check if dest directory already exists
cb => {
if (req.files && req.files.length > 0) {
fs.stat(destPath, (err, stat) => {
if (err && err.code === 'ENOENT') cb();
else cb(new Error(`Directory exists (should not have happened: ${err.code})`));
});
} else {
cb();
}
},
// Unzips zip URL to tmp/<uuid>/ (if any)
cb => {
if (req.body.zipurl) {
let archive = "zipurl.zip";
upload.storage.getDestination(req, archive, (err, dstPath) => {
if (err) cb(err);
else{
let archiveDestPath = path.join(dstPath, archive);
download(req.body.zipurl, archiveDestPath, cb);
}
});
} else {
cb();
}
},
// Move all uploads to data/<uuid>/images dir (if any)
cb => fs.mkdir(destPath, undefined, cb),
cb => fs.mkdir(destGcpPath, undefined, cb),
cb => mv(srcPath, destImagesPath, cb),
cb => {
// Find any *.zip file and extract
fs.readdir(destImagesPath, (err, entries) => {
if (err) cb(err);
else {
async.eachSeries(entries, (entry, cb) => {
if (/\.zip$/gi.test(entry)) {
let filesCount = 0;
fs.createReadStream(path.join(destImagesPath, entry)).pipe(unzip.Parse())
.on('entry', function(entry) {
if (entry.type === 'File') {
filesCount++;
entry.pipe(fs.createWriteStream(path.join(destImagesPath, path.basename(entry.path))));
} else {
entry.autodrain();
}
})
.on('close', () => {
// Verify max images limit
if (config.maxImages && filesCount > config.maxImages) cb(`${filesCount} images uploaded, but this node can only process up to ${config.maxImages}.`);
else cb();
})
.on('error', cb);
} else cb();
}, cb);
}
});
},
cb => {
// Find any *.txt (GCP) file and move it to the data/<uuid>/gcp directory
// also remove any lingering zipurl.zip
fs.readdir(destImagesPath, (err, entries) => {
if (err) cb(err);
else {
async.eachSeries(entries, (entry, cb) => {
if (/\.txt$/gi.test(entry)) {
mv(path.join(destImagesPath, entry), path.join(destGcpPath, entry), cb);
}else if (/\.zip$/gi.test(entry)){
fs.unlink(path.join(destImagesPath, entry), cb);
} else cb();
}, cb);
}
});
},
// Create task
cb => {
new Task(req.id, req.body.name, (err, task) => {
if (err) cb(err);
else {
TaskManager.singleton().addNew(task);
res.json({ uuid: req.id });
cb();
}
}, req.body.options,
req.body.webhook,
req.body.skipPostProcessing === 'true');
}
], err => {
if (err) die(err.message);
});
}
}
}

4
libs/utils.js
Wyświetl plik

@ -13,5 +13,9 @@ module.exports = {
}
}
return defaultValue;
},
sanitize: function(filePath){
return filePath.replace(/(\/|\\)/g, "_");
}
};

2
package.json
Wyświetl plik

@ -1,6 +1,6 @@
{
"name": "node-opendronemap",
"version": "1.3.1",
"version": "1.4.0",
"description": "REST API to access ODM",
"main": "index.js",
"scripts": {

73
public/index.html
Wyświetl plik

@ -14,8 +14,19 @@
padding-top: 50px;
padding-bottom: 20px;
}
.navbar{
background-color: #3498db;
}
a:hover, a:focus, a:active, a{
color: #3498db;
}
#images{
font-weight: bold;
}
#btnSelectFiles, #images{
display: inline-block;
}
</style>
<link href="css/fileinput.css" media="all" rel="stylesheet" type="text/css" />
<link rel="stylesheet" href="css/main.css">
<script src="js/vendor/modernizr-2.8.3.min.js"></script>
@ -29,34 +40,48 @@
<div class="container">
<div class="navbar-header">
<a class="navbar-brand" href="/">NodeODM</a>
<p class="navbar-text navbar-right">Open Source Drone Aerial Imagery Processing</a>
</p>
</div>
</div>
</nav>
<div class="container">
<!-- Example row of columns -->
<div class="row">
<div class="col-md-5">
<h2>New Task</h2>
<br/>
<form enctype="multipart/form-data" onsubmit="return false;">
<div class="form-group form-inline">
<label for="taskName">Project Name:</lable> <input type="text" class="form-control" value="" id="taskName" />
<div id="app">
<div class="form-group form-inline">
<label for="taskName">Project Name:</lable> <input type="text" class="form-control" value="" id="taskName" data-bind="attr: {disabled: uploading()}" />
</div>
<div id="imagesInput" class="form-group" data-bind="visible: mode() === 'file'">
<div id="images">Images and GCP File (optional):</div> <button id="btnSelectFiles" class="btn btn-default btn-sm" data-bind="attr: {disabled: uploading()}">Add Files...</button>
<div data-bind="visible: filesCount() && !uploading()">Selected files: <span data-bind="text: filesCount()"></span></div>
<div data-bind="visible: uploading()" class="progress" style="margin-top: 12px;">
<div class="progress-bar progress-bar-success" role="progressbar" data-bind="text: uploadedFiles() + ' / ' + filesCount() + ' files', style: {width: uploadedPercentage()}">
</div>
</div>
<div data-bind="visible: uploading()" style="min-height: 230px;">
<div data-bind="foreach: fileUploadStatus.items">
<div class="progress">
<div class="progress-bar progress-bar-info" role="progressbar" data-bind="text: key() + ': ' + parseInt(value()) + '%', style: {width: value() + '%'}"></div>
</div>
</div>
</div>
</div>
<div id="zipFileInput" class="form-group" data-bind="visible: mode() === 'url'">
<label for="zipurl">URL to zip file with Images and GCP File (optional):</label> <input id="zipurl" name="zipurl" class="form-control" type="text" data-bind="attr: {disabled: uploading()}" >
<div data-bind="visible: uploading()">
Uploading...
</div>
</div>
<div id="errorBlock" data-bind="visible: error().length > 0, click: dismissError">⚠️ <span data-bind="text: error"></span></div>
<hr/>
<div class="text-right">
<input type="button" class="btn btn-info" data-bind="visible: mode() === 'file', click: toggleMode" value="Switch to URL" />
<input type="button" class="btn btn-info" data-bind="visible: mode() === 'url', click: toggleMode" value="Switch to File Upload" />
<input type="submit" class="btn btn-success" data-bind="attr: {disabled: uploading()}, value: uploading() ? 'Uploading...' : 'Start Task', click: startTask" />
</div>
</div>
<div id="imagesInput" class="form-group">
<label for="images">Aerial Images and GCP List (optional):</label> <input id="images" name="images" multiple accept="image/*|.txt|.zip" type="file">
</div>
<div id="zipFileInput" class="form-group hidden">
<label for="zipurl">URL to zip file with Aerial Images and GCP List (optional):</label> <input id="zipurl" name="zipurl" class="form-control" type="text">
</div>
<div id="errorBlock" class="help-block"></div>
<div class="text-right"><input type="button" class="btn btn-info" value="Switch to URL" id="btnShowImport" />
<input type="button" class="btn btn-info hidden" value="Switch to File Upload" id="btnShowUpload" />
<input type="submit" class="btn btn-success" value="Start Task" id="btnUpload" />
<input type="submit" class="btn btn-success hidden" value="Get ZIP" id="btnImport" /></div>
<div id="options">
<div class="form-inline form-group form-horizontal">
<div data-bind="visible: error(), text: error()" class="alert alert-warning" role="alert"></div>
@ -110,7 +135,6 @@
</form>
</div>
<div class="col-md-7" id="taskList">
<h2>Current Tasks (<span data-bind="text: tasks().length"></span>)</h2>
<p data-bind="visible: tasks().length === 0">No running tasks.</p>
<div data-bind="foreach: tasks">
<div class="task" data-bind="css: {pulsePositive: info().status && info().status.code === 40, pulseNegative: info().status && info().status.code === 30}">
@ -168,8 +192,7 @@
<hr>
<footer>
<p>Links: <a href="https://github.com/OpenDroneMap/NodeODM/blob/master/docs/index.adoc" target="_blank">API Docs</a> | <a href="https://github.com/OpenDroneMap/WebODM" target="_blank">WebODM</a>
<p>This software is released under the terms of the <a href="https://www.gnu.org/licenses/gpl-3.0.en.html" target="_blank">GPLv3 License</a>. See <a href="https://github.com/OpenDroneMap/NodeODM" target="_blank">NodeODM</a> on Github for more information.</p>
<p>This software is released under the terms of the <a href="https://www.gnu.org/licenses/gpl-3.0.en.html" target="_blank">GPLv3 License</a>. See <a href="https://github.com/OpenDroneMap/NodeODM" target="_blank">NodeODM</a> on Github for more information.</p>
</footer>
</div>
<!-- /container -->
@ -179,8 +202,8 @@
</script>
<script src="js/vendor/bootstrap.min.js"></script>
<script src="js/vendor/knockout-3.4.0.js"></script>
<script src="js/fileinput.js" type="text/javascript"></script>
<script src="js/vendor/ko.observableDictionary.js"></script>
<script src="js/dropzone.js" type="text/javascript"></script>
<script src="js/main.js"></script>
</body>

3530
public/js/dropzone.js 100644
Wyświetl plik

Plik diff jest za duży Load Diff

3263
public/js/fileinput.js
Wyświetl plik

Plik diff jest za duży Load Diff

12
public/js/fileinput.min.js vendored
Wyświetl plik

File diff suppressed because one or more lines are too long

236
public/js/main.js
Wyświetl plik

@ -16,6 +16,167 @@ You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
$(function() {
function App(){
this.mode = ko.observable("file");
this.filesCount = ko.observable(0);
this.error = ko.observable("");
this.uploading = ko.observable(false);
this.uuid = ko.observable("");
this.uploadedFiles = ko.observable(0);
this.fileUploadStatus = new ko.observableDictionary({});
this.uploadedPercentage = ko.pureComputed(function(){
return ((this.uploadedFiles() / this.filesCount()) * 100.0) + "%";
}, this);
}
App.prototype.toggleMode = function(){
if (this.mode() === 'file') this.mode('url');
else this.mode('file');
};
App.prototype.dismissError = function(){
this.error("");
};
App.prototype.resetUpload = function(){
this.filesCount(0);
this.error("");
this.uploading(false);
this.uuid("");
this.uploadedFiles(0);
this.fileUploadStatus.removeAll();
dz.removeAllFiles(true);
};
App.prototype.startTask = function(){
var self = this;
this.uploading(true);
this.error("");
this.uuid("");
var die = function(err){
self.error(err);
self.uploading(false);
};
// validate webhook if exists
var webhook = $("#webhook").val();
var regex = new RegExp("^(http[s]?:\\/\\/(www\\.)?|ftp:\\/\\/(www\\.)?|www\\.){1}([0-9A-Za-z-\\.@:%_\+~#=]+)+((\\.[a-zA-Z]{2,3})+)(/(.)*)?(\\?(.)*)?");
if (webhook.length > 0 && !regex.test(webhook)){
die("Invalid webhook URL");
return;
}
// Start upload
var formData = new FormData();
formData.append("name", $("#taskName").val());
formData.append("webhook", $("#webhook").val());
formData.append("skipPostProcessing", !$("#doPostProcessing").prop('checked'));
formData.append("options", JSON.stringify(optionsModel.getUserOptions()));
if (this.mode() === 'file'){
if (this.filesCount() > 0){
$.ajax("/task/new/init?token=" + token, {
type: "POST",
data: formData,
processData: false,
contentType: false
}).done(function(result){
if (result.uuid){
self.uuid(result.uuid);
dz.processQueue();
}else{
die(result.error || result);
}
}).fail(function(){
die("Cannot start task. Is the server available and are you connected to the internet?");
});
}else{
die("No files selected");
}
} else if (this.mode() === 'url'){
this.uploading(true);
formData.append("zipurl", $("#zipurl").val());
$.ajax("/task/new?token=" + token, {
type: "POST",
data: formData,
processData: false,
contentType: false
}).done(function(json){
if (json.uuid){
taskList.add(new Task(json.uuid));
self.resetUpload();
}else{
die(json.error || result);
}
}).fail(function(){
die("Cannot start task. Is the server available and are you connected to the internet?");
});
}
}
Dropzone.autoDiscover = false;
var dz = new Dropzone("div#images", {
paramName: function(){ return "images"; },
url : "/task/new/upload/",
parallelUploads: 8, // http://blog.olamisan.com/max-parallel-http-connections-in-a-browser max parallel connections
uploadMultiple: false,
acceptedFiles: "image/*,text/*",
autoProcessQueue: false,
createImageThumbnails: false,
previewTemplate: '<div style="display:none"></div>',
clickable: document.getElementById("btnSelectFiles"),
chunkSize: 2147483647,
timeout: 2147483647
});
dz.on("processing", function(file){
this.options.url = '/task/new/upload/' + app.uuid() + "?token=" + token;
app.fileUploadStatus.set(file.name, 0);
})
.on("error", function(file){
// Retry
console.log("Error uploading ", file, " put back in queue...");
app.error("Upload of " + file.name + " failed, retrying...");
file.status = Dropzone.QUEUED;
app.fileUploadStatus.remove(file.name);
dz.processQueue();
})
.on("uploadprogress", function(file, progress){
app.fileUploadStatus.set(file.name, progress);
})
.on("addedfiles", function(files){
app.filesCount(app.filesCount() + files.length);
})
.on("complete", function(file){
if (file.status === "success"){
app.uploadedFiles(app.uploadedFiles() + 1);
}
app.fileUploadStatus.remove(file.name);
dz.processQueue();
})
.on("queuecomplete", function(files){
// Commit
$.ajax("/task/new/commit/" + app.uuid() + "?token=" + token, {
type: "POST",
}).done(function(json){
if (json.uuid){
taskList.add(new Task(json.uuid));
app.resetUpload();
}else{
app.error(json.error || json);
}
app.uploading(false);
}).fail(function(){
app.error("Cannot commit task. Is the server available and are you connected to the internet?");
app.uploading(false);
});
})
.on("reset", function(){
app.filesCount(0);
});
app = new App();
ko.applyBindings(app, document.getElementById('app'));
function query(key) {
key = key.replace(/[*+?^$.\[\]{}()|\\\/]/g, "\\$&"); // escape RegEx meta chars
var match = location.search.match(new RegExp("[?&]"+key+"=([^&]+)(&|$)"));
@ -285,44 +446,6 @@ $(function() {
var taskList = new TaskList();
ko.applyBindings(taskList, document.getElementById('taskList'));
// Handle uploads
$("#images").fileinput({
uploadUrl: '/task/new?token=' + token,
showPreview: false,
allowedFileExtensions: ['jpg', 'jpeg', 'txt', 'zip'],
elErrorContainer: '#errorBlock',
showUpload: false,
uploadAsync: false,
// ajaxSettings: { headers: { 'set-uuid': '8366b2ad-a608-4cd1-bdcb-c3d84a034623' } },
uploadExtraData: function() {
return {
name: $("#taskName").val(),
zipurl: $("#zipurl").val(),
webhook: $("#webhook").val(),
skipPostProcessing: !$("#doPostProcessing").prop('checked'),
options: JSON.stringify(optionsModel.getUserOptions())
};
}
});
$("#btnUpload").click(function() {
$("#btnUpload").attr('disabled', true)
.val("Uploading...");
// validate webhook if exists
var webhook = $("#webhook").val();
var regex = new RegExp("^(http[s]?:\\/\\/(www\\.)?|ftp:\\/\\/(www\\.)?|www\\.){1}([0-9A-Za-z-\\.@:%_\+~#=]+)+((\\.[a-zA-Z]{2,3})+)(/(.)*)?(\\?(.)*)?");
if(webhook.length > 0 && !regex.test(webhook)){
$('#errorBlock').text("Webhook url is not valid..");
$("#btnUpload").attr('disabled', false).val("Start Task");
return;
}
// Start upload
$("#images").fileinput('upload');
});
$('#resetWebhook').on('click', function(){
$("#webhook").val('');
});
@ -331,43 +454,6 @@ $(function() {
$("#doPostProcessing").prop('checked', false);
});
// zip file control
$('#btnShowImport').on('click', function(e){
e.preventDefault();
$('#zipFileInput').removeClass('hidden');
$('#btnShowUpload').removeClass('hidden');
$('#imagesInput').addClass('hidden');
$('#btnShowImport').addClass('hidden');
});
$('#btnShowUpload').on('click', function(e){
e.preventDefault();
$('#imagesInput').removeClass('hidden');
$('#btnShowImport').removeClass('hidden');
$('#zipFileInput').addClass('hidden');
$('#btnShowUpload').addClass('hidden');
$('#zipurl').val('');
});
var btnUploadLabel = $("#btnUpload").val();
$("#images")
.on('filebatchuploadsuccess', function(e, params) {
$("#images").fileinput('reset');
if (params.response && params.response.uuid) {
taskList.add(new Task(params.response.uuid));
}
})
.on('filebatchuploadcomplete', function() {
$("#btnUpload").removeAttr('disabled')
.val(btnUploadLabel);
})
.on('filebatchuploaderror', console.warn);
// Load options
function Option(properties) {
this.properties = properties;

224
public/js/vendor/ko.observableDictionary.js vendored 100644
Wyświetl plik

@ -0,0 +1,224 @@
// Knockout Observable Dictionary
// (c) James Foster
// License: MIT (http://www.opensource.org/licenses/mit-license.php)
(function () {
function DictionaryItem(key, value, dictionary) {
var observableKey = new ko.observable(key);
this.value = new ko.observable(value);
this.key = new ko.computed({
read: observableKey,
write: function (newKey) {
var current = observableKey();
if (current == newKey) return;
// no two items are allowed to share the same key.
dictionary.remove(newKey);
observableKey(newKey);
}
});
}
ko.observableDictionary = function (dictionary, keySelector, valueSelector) {
var result = {};
result.items = new ko.observableArray();
result._wrappers = {};
result._keySelector = keySelector || function (value, key) { return key; };
result._valueSelector = valueSelector || function (value) { return value; };
if (typeof keySelector == 'string') result._keySelector = function (value) { return value[keySelector]; };
if (typeof valueSelector == 'string') result._valueSelector = function (value) { return value[valueSelector]; };
ko.utils.extend(result, ko.observableDictionary['fn']);
result.pushAll(dictionary);
return result;
};
ko.observableDictionary['fn'] = {
remove: function (valueOrPredicate) {
var predicate = valueOrPredicate;
if (valueOrPredicate instanceof DictionaryItem) {
predicate = function (item) {
return item.key() === valueOrPredicate.key();
};
}
else if (typeof valueOrPredicate != "function") {
predicate = function (item) {
return item.key() === valueOrPredicate;
};
}
ko.observableArray['fn'].remove.call(this.items, predicate);
},
push: function (key, value) {
var item = null;
if (key instanceof DictionaryItem) {
// handle the case where only a DictionaryItem is passed in
item = key;
value = key.value();
key = key.key();
}
if (value === undefined) {
value = this._valueSelector(key);
key = this._keySelector(value);
}
else {
value = this._valueSelector(value);
}
var current = this.get(key, false);
if (current) {
// update existing value
current(value);
return current;
}
if (!item) {
item = new DictionaryItem(key, value, this);
}
ko.observableArray['fn'].push.call(this.items, item);
return value;
},
pushAll: function (dictionary) {
var self = this;
var items = self.items();
if (dictionary instanceof Array) {
$.each(dictionary, function (index, item) {
var key = self._keySelector(item, index);
var value = self._valueSelector(item);
items.push(new DictionaryItem(key, value, self));
});
}
else {
for (var prop in dictionary) {
if (dictionary.hasOwnProperty(prop)) {
var item = dictionary[prop];
var key = self._keySelector(item, prop);
var value = self._valueSelector(item);
items.push(new DictionaryItem(key, value, self));
}
}
}
self.items.valueHasMutated();
},
sort: function (method) {
if (method === undefined) {
method = function (a, b) {
return defaultComparison(a.key(), b.key());
};
}
return ko.observableArray['fn'].sort.call(this.items, method);
},
indexOf: function (key) {
if (key instanceof DictionaryItem) {
return ko.observableArray['fn'].indexOf.call(this.items, key);
}
var underlyingArray = this.items();
for (var index = 0; index < underlyingArray.length; index++) {
if (underlyingArray[index].key() == key)
return index;
}
return -1;
},
get: function (key, wrap) {
if (wrap == false)
return getValue(key, this.items());
var wrapper = this._wrappers[key];
if (wrapper == null) {
wrapper = this._wrappers[key] = new ko.computed({
read: function () {
var value = getValue(key, this.items());
return value ? value() : null;
},
write: function (newValue) {
var value = getValue(key, this.items());
if (value)
value(newValue);
else
this.push(key, newValue);
}
}, this);
}
return wrapper;
},
set: function (key, value) {
return this.push(key, value);
},
keys: function () {
return ko.utils.arrayMap(this.items(), function (item) { return item.key(); });
},
values: function () {
return ko.utils.arrayMap(this.items(), function (item) { return item.value(); });
},
removeAll: function () {
this.items.removeAll();
},
toJSON: function () {
var result = {};
var items = ko.utils.unwrapObservable(this.items);
ko.utils.arrayForEach(items, function (item) {
var key = ko.utils.unwrapObservable(item.key);
var value = ko.utils.unwrapObservable(item.value);
result[key] = value;
});
return result;
}
};
function getValue(key, items) {
var found = ko.utils.arrayFirst(items, function (item) {
return item.key() == key;
});
return found ? found.value : null;
}
})();
// Utility methods
// ---------------------------------------------
function isNumeric(n) {
return !isNaN(parseFloat(n)) && isFinite(n);
}
function defaultComparison(a, b) {
if (isNumeric(a) && isNumeric(b)) return a - b;
a = a.toString();
b = b.toString();
return a == b ? 0 : (a < b ? -1 : 1);
}
// ---------------------------------------------