Merge pull request #571 from pierotofy/improvements

Improvements
pull/579/head
Piero Toffanin 2018-12-10 20:43:55 -05:00 zatwierdzone przez GitHub
commit 3714f438b6
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
43 zmienionych plików z 874 dodań i 175 usunięć

6
.gitmodules vendored
Wyświetl plik

@ -1,3 +1,3 @@
[submodule "nodeodm/external/node-OpenDroneMap"]
path = nodeodm/external/node-OpenDroneMap
url = https://github.com/OpenDroneMap/node-OpenDroneMap
[submodule "nodeodm/external/NodeODM"]
path = nodeodm/external/NodeODM
url = https://github.com/OpenDroneMap/NodeODM

Wyświetl plik

@ -32,7 +32,7 @@ RUN ln -s /webodm/nginx/crontab /etc/cron.d/nginx-cron && chmod 0644 /webodm/ngi
RUN git submodule update --init
WORKDIR /webodm/nodeodm/external/node-OpenDroneMap
WORKDIR /webodm/nodeodm/external/NodeODM
RUN npm install --quiet
WORKDIR /webodm

Wyświetl plik

@ -82,7 +82,7 @@ You can also run WebODM from a Live USB/DVD. See [LiveODM](https://www.opendrone
### Add More Processing Nodes
WebODM can be linked to one or more processing nodes running [node-OpenDroneMap](https://github.com/OpenDroneMap/node-OpenDroneMap). The default configuration already includes a "node-odm-1" processing node which runs on the same machine as WebODM, just to help you get started. As you become more familiar with WebODM, you might want to install processing nodes on separate machines.
WebODM can be linked to one or more processing nodes running [NodeODM](https://github.com/OpenDroneMap/NodeODM). The default configuration already includes a "node-odm-1" processing node which runs on the same machine as WebODM, just to help you get started. As you become more familiar with WebODM, you might want to install processing nodes on separate machines.
Adding more processing nodes will allow you to run multiple jobs in parallel.
@ -123,7 +123,7 @@ While starting WebODM you get: `'WaitNamedPipe','The system cannot find the file
While Accessing the WebODM interface you get: `OperationalError at / could not translate host name “db” to address: Name or service not known` or `ProgrammingError at / relation “auth_user” does not exist` | Try restarting your computer, then type: `./webodm.sh restart`
Task output or console shows one of the following:<ul><li>`MemoryError`</li><li>`Killed`</li></ul> | Make sure that your Docker environment has enough RAM allocated: [MacOS Instructions](http://stackoverflow.com/a/39720010), [Windows Instructions](https://docs.docker.com/docker-for-windows/#advanced)
After an update, you get: `django.contrib.auth.models.DoesNotExist: Permission matching query does not exist.` | Try to remove your WebODM folder and start from a fresh git clone
Task fails with `Process exited with code null`, no task console output - OR - console output shows `Illegal Instruction` - OR - console output shows `Child returned 132` | If the computer running node-opendronemap is using an old or 32bit CPU, you need to compile [OpenDroneMap](https://github.com/OpenDroneMap/OpenDroneMap) from sources and setup node-opendronemap natively. You cannot use docker. Docker images work with CPUs with 64-bit extensions, MMX, SSE, SSE2, SSE3 and SSSE3 instruction set support or higher.
Task fails with `Process exited with code null`, no task console output - OR - console output shows `Illegal Instruction` - OR - console output shows `Child returned 132` | If the computer running NodeODM is using an old or 32bit CPU, you need to compile [OpenDroneMap](https://github.com/OpenDroneMap/OpenDroneMap) from sources and setup NodeODM natively. You cannot use docker. Docker images work with CPUs with 64-bit extensions, MMX, SSE, SSE2, SSE3 and SSSE3 instruction set support or higher.
On Windows, docker-compose fails with `Failed to execute the script docker-compose` | Make sure you have enabled VT-x virtualization in the BIOS
Cannot access WebODM using Microsoft Edge on Windows 10 | Try to tweak your internet properties according to [these instructions](http://www.hanselman.com/blog/FixedMicrosoftEdgeCantSeeOrOpenVirtualBoxhostedLocalWebSites.aspx)
Getting a `No space left on device` error, but hard drive has enough space left | Docker on Windows by default will allocate only 20GB of space to the default docker-machine. You need to increase that amount. See [this link](http://support.divio.com/local-development/docker/managing-disk-space-in-your-docker-vm) and [this link](https://www.howtogeek.com/124622/how-to-enlarge-a-virtual-machines-disk-in-virtualbox-or-vmware/)
@ -201,7 +201,7 @@ The [OpenDroneMap project](https://github.com/OpenDroneMap/) is composed of seve
- [ODM](https://github.com/OpenDroneMap/ODM) is a command line toolkit that processes aerial images. Users comfortable with the command line are probably OK using this component alone.
- [NodeODM](https://github.com/OpenDroneMap/NodeODM) is a lightweight interface and API (Application Program Interface) built directly on top of [ODM](https://github.com/OpenDroneMap/ODM). Users not comfortable with the command line can use this interface to process aerial images and developers can use the API to build applications. Features such as user authentication, map displays, etc. are not provided.
- [WebODM](https://github.com/OpenDroneMap/WebODM) adds more features such as user authentication, map displays, 3D displays, a higher level API and the ability to orchestrate multiple processing nodes (run jobs in parallel). Processing nodes are simply servers running [NodeODM](https://github.com/OpenDroneMap/node-OpenDroneMap).
- [WebODM](https://github.com/OpenDroneMap/WebODM) adds more features such as user authentication, map displays, 3D displays, a higher level API and the ability to orchestrate multiple processing nodes (run jobs in parallel). Processing nodes are simply servers running [NodeODM](https://github.com/OpenDroneMap/NodeODM).
![webodm](https://cloud.githubusercontent.com/assets/1951843/25567386/5aeec7aa-2dba-11e7-9169-aca97b70db79.png)
@ -281,7 +281,7 @@ WebODM is built with scalability and performance in mind. While the default setu
![Architecture](https://user-images.githubusercontent.com/1951843/36916884-3a269a7a-1e23-11e8-997a-a57cd6ca7950.png)
A few things to note:
* We use Celery workers to do background tasks such as resizing images and processing task results, but we use an ad-hoc scheduling mechanism to communicate with node-OpenDroneMap (which processes the orthophotos, 3D models, etc.). The choice to use two separate systems for task scheduling is due to the flexibility that an ad-hoc mechanism gives us for certain operations (capture task output, persistent data and ability to restart tasks mid-way, communication via REST calls, etc.).
* We use Celery workers to do background tasks such as resizing images and processing task results, but we use an ad-hoc scheduling mechanism to communicate with NodeODM (which processes the orthophotos, 3D models, etc.). The choice to use two separate systems for task scheduling is due to the flexibility that an ad-hoc mechanism gives us for certain operations (capture task output, persistent data and ability to restart tasks mid-way, communication via REST calls, etc.).
* If loaded on multiple machines, Celery workers should all share their `app/media` directory with the Django application (via network shares). You can manage workers via `./worker.sh`

Wyświetl plik

@ -30,7 +30,7 @@ class ProcessingNodeFilter(FilterSet):
class Meta:
model = ProcessingNode
fields = ['has_available_options', 'id', 'hostname', 'port', 'api_version', 'queue_count', ]
fields = ['has_available_options', 'id', 'hostname', 'port', 'api_version', 'queue_count', 'max_images', ]
class ProcessingNodeViewSet(viewsets.ModelViewSet):
"""

Wyświetl plik

@ -24,9 +24,16 @@ class TaskIDsSerializer(serializers.BaseSerializer):
class TaskSerializer(serializers.ModelSerializer):
project = serializers.PrimaryKeyRelatedField(queryset=models.Project.objects.all())
processing_node = serializers.PrimaryKeyRelatedField(queryset=ProcessingNode.objects.all())
processing_node_name = serializers.SerializerMethodField()
images_count = serializers.SerializerMethodField()
can_rerun_from = serializers.SerializerMethodField()
def get_processing_node_name(self, obj):
if obj.processing_node is not None:
return str(obj.processing_node)
else:
return None
def get_images_count(self, obj):
return obj.imageupload_set.count()
@ -39,7 +46,7 @@ class TaskSerializer(serializers.ModelSerializer):
TODO: this could be improved by returning an empty array if a task was created
and purged by the processing node (which would require knowing how long a task is being kept
see https://github.com/OpenDroneMap/node-OpenDroneMap/issues/32
see https://github.com/OpenDroneMap/NodeODM/issues/32
:return: array of valid rerun-from parameters
"""
if obj.processing_node is not None:

Wyświetl plik

@ -0,0 +1,27 @@
# Generated by Django 2.0.3 on 2018-12-05 16:44
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('app', '0021_auto_20180726_1746'),
]
operations = [
migrations.RemoveField(
model_name='task',
name='ground_control_points',
),
migrations.AddField(
model_name='task',
name='resize_progress',
field=models.FloatField(blank=True, default=0.0, help_text="Value between 0 and 1 indicating the resize progress of this task's images"),
),
migrations.AddField(
model_name='task',
name='upload_progress',
field=models.FloatField(blank=True, default=0.0, help_text="Value between 0 and 1 indicating the upload progress of this task's files to the processing node"),
),
]

Wyświetl plik

@ -0,0 +1,18 @@
# Generated by Django 2.0.3 on 2018-12-07 18:53
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('app', '0022_auto_20181205_1644'),
]
operations = [
migrations.AddField(
model_name='task',
name='running_progress',
field=models.FloatField(blank=True, default=0.0, help_text="Value between 0 and 1 indicating the running progress (estimated) of this task"),
),
]

Wyświetl plik

@ -2,6 +2,7 @@ import logging
import os
import shutil
import zipfile
import time
import uuid as uuid_module
import json
@ -69,7 +70,13 @@ def validate_task_options(value):
def resize_image(image_path, resize_to):
def resize_image(image_path, resize_to, done=None):
"""
:param image_path: path to the image
:param resize_to: target size to resize this image to (largest side)
:param done: optional callback
:return: path and resize ratio
"""
try:
im = Image.open(image_path)
path, ext = os.path.splitext(image_path)
@ -105,9 +112,16 @@ def resize_image(image_path, resize_to):
logger.info("Resized {} to {}x{}".format(image_path, resized_width, resized_height))
except IOError as e:
logger.warning("Cannot resize {}: {}.".format(image_path, str(e)))
if done is not None:
done()
return None
return {'path': image_path, 'resize_ratio': ratio}
retval = {'path': image_path, 'resize_ratio': ratio}
if done is not None:
done(retval)
return retval
class Task(models.Model):
ASSETS_MAP = {
@ -142,6 +156,22 @@ class Task(models.Model):
(pending_actions.RESIZE, 'RESIZE'),
)
# Not an exact science
TASK_OUTPUT_MILESTONES = {
'Running ODM Load Dataset Cell': 0.01,
'Running ODM Load Dataset Cell - Finished': 0.05,
'opensfm/bin/opensfm match_features': 0.10,
'opensfm/bin/opensfm reconstruct': 0.20,
'opensfm/bin/opensfm export_visualsfm': 0.30,
'Running ODM Meshing Cell': 0.60,
'Running MVS Texturing Cell': 0.65,
'Running ODM Georeferencing Cell': 0.70,
'Running ODM DEM Cell': 0.80,
'Running ODM Orthophoto Cell': 0.85,
'Running ODM OrthoPhoto Cell - Finished': 0.90,
'Compressing all.zip:': 0.95
}
id = models.UUIDField(primary_key=True, default=uuid_module.uuid4, unique=True, serialize=False, editable=False)
uuid = models.CharField(max_length=255, db_index=True, default='', blank=True, help_text="Identifier of the task (as returned by OpenDroneMap's REST API)")
@ -155,7 +185,6 @@ class Task(models.Model):
options = fields.JSONField(default=dict(), blank=True, help_text="Options that are being used to process this task", validators=[validate_task_options])
available_assets = fields.ArrayField(models.CharField(max_length=80), default=list(), blank=True, help_text="List of available assets to download")
console_output = models.TextField(null=False, default="", blank=True, help_text="Console output of the OpenDroneMap's process")
ground_control_points = models.FileField(null=True, blank=True, upload_to=gcp_directory_path, help_text="Optional Ground Control Points file to use for processing")
orthophoto_extent = GeometryField(null=True, blank=True, srid=4326, help_text="Extent of the orthophoto created by OpenDroneMap")
dsm_extent = GeometryField(null=True, blank=True, srid=4326, help_text="Extent of the DSM created by OpenDroneMap")
@ -168,6 +197,15 @@ class Task(models.Model):
public = models.BooleanField(default=False, help_text="A flag indicating whether this task is available to the public")
resize_to = models.IntegerField(default=-1, help_text="When set to a value different than -1, indicates that the images for this task have been / will be resized to the size specified here before processing.")
upload_progress = models.FloatField(default=0.0,
help_text="Value between 0 and 1 indicating the upload progress of this task's files to the processing node",
blank=True)
resize_progress = models.FloatField(default=0.0,
help_text="Value between 0 and 1 indicating the resize progress of this task's images",
blank=True)
running_progress = models.FloatField(default=0.0,
help_text="Value between 0 and 1 indicating the running progress (estimated) of this task",
blank=True)
def __init__(self, *args, **kwargs):
super(Task, self).__init__(*args, **kwargs)
@ -289,6 +327,7 @@ class Task(models.Model):
try:
if self.pending_action == pending_actions.RESIZE:
resized_images = self.resize_images()
self.refresh_from_db()
self.resize_gcp(resized_images)
self.pending_action = None
self.save()
@ -333,8 +372,17 @@ class Task(models.Model):
images = [image.path() for image in self.imageupload_set.all()]
# Track upload progress, but limit the number of DB updates
# to every 2 seconds (and always record the 100% progress)
last_update = 0
def callback(progress):
nonlocal last_update
if time.time() - last_update >= 2 or (progress >= 1.0 - 1e-6 and progress <= 1.0 + 1e-6):
Task.objects.filter(pk=self.id).update(upload_progress=progress)
last_update = time.time()
# This takes a while
uuid = self.processing_node.process_new_task(images, self.name, self.options)
uuid = self.processing_node.process_new_task(images, self.name, self.options, callback)
# Refresh task object before committing change
self.refresh_from_db()
@ -400,12 +448,14 @@ class Task(models.Model):
# We also remove the "rerun-from" parameter if it's set
self.options = list(filter(lambda d: d['name'] != 'rerun-from', self.options))
self.upload_progress = 0
self.console_output = ""
self.processing_time = -1
self.status = None
self.last_error = None
self.pending_action = None
self.running_progress = 0
self.save()
else:
raise ProcessingError("Cannot restart a task that has no processing node")
@ -439,7 +489,14 @@ class Task(models.Model):
current_lines_count = len(self.console_output.split("\n"))
console_output = self.processing_node.get_task_console_output(self.uuid, current_lines_count)
if len(console_output) > 0:
self.console_output += console_output + '\n'
self.console_output += "\n".join(console_output) + '\n'
# Update running progress
for line in console_output:
for line_match, value in self.TASK_OUTPUT_MILESTONES.items():
if line_match in line:
self.running_progress = value
break
if "errorMessage" in info["status"]:
self.last_error = info["status"]["errorMessage"]
@ -498,6 +555,7 @@ class Task(models.Model):
logger.info("Populated extent field with {} for {}".format(raster_path, self))
self.update_available_assets_field()
self.running_progress = 1.0
self.save()
from app.plugins import signals as plugin_signals
@ -620,12 +678,27 @@ class Task(models.Model):
return []
images_path = self.find_all_files_matching(r'.*\.jpe?g$')
total_images = len(images_path)
resized_images_count = 0
last_update = 0
def callback(retval=None):
nonlocal last_update
nonlocal resized_images_count
nonlocal total_images
resized_images_count += 1
if time.time() - last_update >= 2:
Task.objects.filter(pk=self.id).update(resize_progress=(float(resized_images_count) / float(total_images)))
last_update = time.time()
with ThreadPoolExecutor(max_workers=cpu_count()) as executor:
resized_images = list(filter(lambda i: i is not None, executor.map(
partial(resize_image, resize_to=self.resize_to),
partial(resize_image, resize_to=self.resize_to, done=callback),
images_path)))
Task.objects.filter(pk=self.id).update(resize_progress=1.0)
return resized_images
def resize_gcp(self, resized_images):

Wyświetl plik

@ -264,4 +264,8 @@ footer{
.full-height{
height: calc(100vh - 110px);
padding-bottom: 12px;
}
.floatfix{
clear: both;
}

Wyświetl plik

@ -131,6 +131,14 @@ a, a:hover, a:focus{
}
}
.theme-color-button-danger{
color: theme("button_danger");
}
.theme-color-button-primary{
color: theme("button_primary");
}
/* Header background */
#navbar-top{
background-color: theme("header_background");
@ -204,7 +212,6 @@ pre.prettyprint,
background-color: theme("failed");
}
/* ModelView.jsx specific */
.model-view #potree_sidebar_container {
.dropdown-menu > li > a{

Wyświetl plik

@ -24,6 +24,8 @@ class Console extends React.Component {
this.setRef = this.setRef.bind(this);
this.handleMouseOver = this.handleMouseOver.bind(this);
this.handleMouseOut = this.handleMouseOut.bind(this);
this.downloadTxt = this.downloadTxt.bind(this);
this.copyTxt = this.copyTxt.bind(this);
}
componentDidMount(){
@ -64,6 +66,7 @@ class Console extends React.Component {
}
downloadTxt(filename="console.txt"){
console.log(filename);
function saveAs(uri, filename) {
let link = document.createElement('a');
if (typeof link.download === 'string') {
@ -93,7 +96,7 @@ class Console extends React.Component {
el.select();
document.execCommand('copy');
document.body.removeChild(el);
console.log("Task output copied to clipboard");
console.log("Output copied to clipboard");
}
tearDownDynamicSource(){
@ -140,22 +143,40 @@ class Console extends React.Component {
}
let i = 0;
return (
<pre className={`console prettyprint
${this.props.lang ? `lang-${this.props.lang}` : ""}
${this.props.lines ? "linenums" : ""}`}
style={{height: (this.props.height ? this.props.height : "auto")}}
onMouseOver={this.handleMouseOver}
onMouseOut={this.handleMouseOut}
ref={this.setRef}
>
{this.state.lines.map(line => {
if (this.props.lang) return (<div key={i++} dangerouslySetInnerHTML={prettyLine(line)}></div>);
else return line + "\n";
})}
{"\n"}
</pre>
);
let lines = this.state.lines;
if (this.props.maximumLines && lines.length > this.props.maximumLines){
lines = lines.slice(-this.props.maximumLines);
lines.unshift(`... output truncated at ${this.props.maximumLines} lines ...`);
}
const items = [
<pre key="console" className={`console prettyprint
${this.props.lang ? `lang-${this.props.lang}` : ""}
${this.props.lines ? "linenums" : ""}
${this.props.className || ""}`}
style={{height: (this.props.height ? this.props.height : "auto")}}
onMouseOver={this.handleMouseOver}
onMouseOut={this.handleMouseOut}
ref={this.setRef}
>{lines.map(line => {
if (this.props.lang) return (<div key={i++} dangerouslySetInnerHTML={prettyLine(line)}></div>);
else return line + "\n";
})}
{"\n"}
</pre>];
if (this.props.showConsoleButtons){
items.push(<div key="buttons" className="console-buttons">
<a href="javascript:void(0);" onClick={() => this.downloadTxt()} className="btn btn-sm btn-primary" title="Download To File">
<i className="fa fa-download"></i>
</a>
<a href="javascript:void(0);" onClick={this.copyTxt} className="btn btn-sm btn-primary" title="Copy To Clipboard">
<i className="fa fa-clipboard"></i>
</a>
</div>);
}
return items;
}
}

Wyświetl plik

@ -0,0 +1,16 @@
const values = {};
export default {
getValue: function(className, property, element = 'div'){
const k = className + '|' + property;
if (values[k]) return values[k];
else{
let d = document.createElement(element);
d.style.display = "none";
d.className = className;
document.body.appendChild(d);
values[k] = getComputedStyle(d)[property];
document.body.removeChild(d);
return values[k];
}
}
}

Wyświetl plik

@ -0,0 +1,54 @@
export default {
get: function(){
return [{
action: "dataset",
label: "Load Dataset",
icon: "fa fa-database",
beginsWith: "Running ODM Load Dataset Cell",
endsWith: "Running ODM Load Dataset Cell - Finished"
},
{
action: "opensfm",
label: "Structure From Motion / MVS",
icon: "fa fa-camera",
beginsWith: "Running ODM OpenSfM Cell",
endsWith: "Running ODM Meshing Cell"
},
{
action: "odm_meshing",
label: "Meshing",
icon: "fa fa-cube",
beginsWith: "Running ODM Meshing Cell",
endsWith: "Running ODM Meshing Cell - Finished"
},
{
action: "mvs_texturing",
label: "Texturing",
icon: "fa fa-connectdevelop",
beginsWith: "Running MVS Texturing Cell",
endsWith: "Running ODM Texturing Cell - Finished"
},
{
action: "odm_georeferencing",
label: "Georeferencing",
icon: "fa fa-globe",
beginsWith: "Running ODM Georeferencing Cell",
endsWith: "Running ODM Georeferencing Cell - Finished"
},
{
action: "odm_dem",
label: "DEM",
icon: "fa fa-area-chart",
beginsWith: "Running ODM DEM Cell",
endsWith: "Running ODM DEM Cell - Finished"
},
{
action: "odm_orthophoto",
label: "Orthophoto",
icon: "fa fa-map-o",
beginsWith: "Running ODM Orthophoto Cell",
endsWith: "Running ODM OrthoPhoto Cell - Finished"
}
];
}
};

Wyświetl plik

@ -45,7 +45,7 @@ class AssetDownloadButtons extends React.Component {
{assetDownloads.map((asset, i) => {
if (!asset.separator){
return (<li key={i}>
<a href="javascript:void(0);" onClick={this.downloadAsset(asset)}><i className={asset.icon}></i> {asset.label}</a>
<a href="javascript:void(0);" onClick={this.downloadAsset(asset)}><i className={asset.icon + " fa-fw"}></i> {asset.label}</a>
</li>);
}else{
return (<li key={i} className="divider"></li>);

Wyświetl plik

@ -0,0 +1,234 @@
import React from 'react';
import PropTypes from 'prop-types';
import '../css/BasicTaskView.scss';
import update from 'immutability-helper';
import PipelineSteps from '../classes/PipelineSteps';
import StatusCodes from '../classes/StatusCodes';
import $ from 'jquery';
class BasicTaskView extends React.Component {
static defaultProps = {};
static propTypes = {
source: PropTypes.oneOfType([
PropTypes.string.isRequired,
PropTypes.func.isRequired
]),
taskStatus: PropTypes.number,
onAddLines: PropTypes.func
};
constructor(props){
super();
this.state = {
lines: [],
currentRf: 0,
rf: PipelineSteps.get(),
loaded: false
};
this.imageUpload = {
action: "imageupload",
label: "Image Resize / Upload",
icon: "fa fa-image"
};
// Add a post processing step
this.state.rf.push({
action: "postprocessing",
label: "Post Processing",
icon: "fa fa-file-archive-o",
beginsWith: "Running ODM OrthoPhoto Cell - Finished",
endsWith: null
});
this.addLines = this.addLines.bind(this);
this.setupDynamicSource = this.setupDynamicSource.bind(this);
this.reset = this.reset.bind(this);
this.tearDownDynamicSource = this.tearDownDynamicSource.bind(this);
this.getRfEndStatus = this.getRfEndStatus.bind(this);
this.getRfRunningStatus = this.getRfRunningStatus.bind(this);
this.suffixFor = this.suffixFor.bind(this);
this.updateRfState = this.updateRfState.bind(this);
this.getInitialStatus = this.getInitialStatus.bind(this);
}
componentDidMount(){
this.reset();
}
setupDynamicSource(){
const updateFromSource = () => {
let sourceUrl = typeof this.props.source === 'function' ?
this.props.source(this.state.lines.length) :
this.props.source;
// Fetch
this.sourceRequest = $.get(sourceUrl, text => {
if (text !== ""){
let lines = text.split("\n");
this.addLines(lines);
}
})
.always((_, textStatus) => {
if (textStatus !== "abort" && this.props.refreshInterval !== undefined){
this.sourceTimeout = setTimeout(updateFromSource, this.props.refreshInterval);
}
if (!this.state.loaded) this.setState({loaded: true});
});
};
updateFromSource();
}
getRfEndStatus(){
return this.props.taskStatus === StatusCodes.COMPLETED ?
'completed' :
'errored';
}
getRfRunningStatus(){
return [StatusCodes.RUNNING, StatusCodes.QUEUED].indexOf(this.props.taskStatus) !== -1 ?
'running' :
'errored';
}
getInitialStatus(){
if ([null, StatusCodes.QUEUED, StatusCodes.RUNNING].indexOf(this.props.taskStatus) !== -1){
return 'queued';
}else{
return this.getRfEndStatus();
}
}
reset(){
this.state.rf.forEach(p => {
p.state = this.getInitialStatus();
});
if ([StatusCodes.RUNNING].indexOf(this.props.taskStatus) !== -1){
this.state.rf[0].state = 'running';
}
this.tearDownDynamicSource();
this.setState({lines: [], currentRf: 0, loaded: false, rf: this.state.rf});
this.setupDynamicSource();
}
tearDownDynamicSource(){
if (this.sourceTimeout) clearTimeout(this.sourceTimeout);
if (this.sourceRequest) this.sourceRequest.abort();
}
componentDidUpdate(prevProps){
let taskFailed;
let taskCompleted;
let taskRestarted;
if (prevProps.taskStatus !== this.props.taskStatus){
taskFailed = [StatusCodes.FAILED, StatusCodes.CANCELED].indexOf(this.props.taskStatus) !== -1;
taskCompleted = this.props.taskStatus === StatusCodes.COMPLETED;
taskRestarted = this.props.taskStatus === null;
}
this.updateRfState(taskFailed, taskCompleted, taskRestarted);
}
componentWillUnmount(){
this.tearDownDynamicSource();
}
addLines(lines){
if (!Array.isArray(lines)) lines = [lines];
let currentRf = this.state.currentRf;
const updateRf = (rfIndex, line) => {
const current = this.state.rf[rfIndex];
if (!current) return;
if (current.beginsWith && line.endsWith(current.beginsWith)){
current.state = this.getRfRunningStatus();
// Set previous as done
if (this.state.rf[rfIndex - 1]){
this.state.rf[rfIndex - 1].state = 'completed';
}
}else if (current.endsWith && line.endsWith(current.endsWith)){
current.state = this.getRfEndStatus();
// Check next
updateRf(rfIndex + 1, line);
currentRf = rfIndex + 1;
}
};
lines.forEach(line => {
updateRf(currentRf, line);
});
this.setState(update(this.state, {
lines: {$push: lines}
}));
this.setState({
rf: this.state.rf,
currentRf
});
this.updateRfState();
if (this.props.onAddLines) this.props.onAddLines(lines);
}
updateRfState(taskFailed, taskCompleted, taskRestarted){
// If the task has just failed, update all items that were either running or in queued state
if (taskFailed){
this.state.rf.forEach(p => {
if (p.state === 'queued' || p.state === 'running') p.state = this.getInitialStatus();
});
}
// If completed, all steps must have completed
if (taskCompleted){
this.state.rf.forEach(p => p.state = 'completed');
}
if (taskRestarted){
this.state.rf.forEach(p => p.state = 'queued');
}
}
suffixFor(state){
if (state === 'running'){
return (<span>...</span>);
}else if (state === 'completed'){
return (<i className="fa fa-check"></i>);
}else if (state === 'errored'){
return (<i className="fa fa-times"></i>);
}
}
render() {
const { rf, loaded } = this.state;
const imageUploadState = this.props.taskStatus === null
? 'running'
: 'completed';
return (<div className={"basic-task-view " + (loaded ? 'loaded' : '')}>
<div className={imageUploadState + " processing-step"}>
<i className={this.imageUpload.icon + " fa-fw"}></i> {this.imageUpload.label} {this.suffixFor(imageUploadState)}
</div>
{rf.map(p => {
const state = loaded ? p.state : 'queued';
return (<div key={p.action} className={state + " processing-step"}>
<i className={p.icon + " fa-fw"}></i> {p.label} {this.suffixFor(state)}
</div>);
})}
</div>);
}
}
export default BasicTaskView;

Wyświetl plik

@ -79,7 +79,8 @@ class ProjectListItem extends React.Component {
progress: 0,
totalCount: 0,
totalBytes: 0,
totalBytesSent: 0
totalBytesSent: 0,
lastUpdated: 0
};
}
@ -152,9 +153,18 @@ class ProjectListItem extends React.Component {
});
this.dz.on("totaluploadprogress", (progress, totalBytes, totalBytesSent) => {
this.setUploadState({
progress, totalBytes, totalBytesSent
});
// Limit updates since this gets called a lot
let now = (new Date()).getTime();
// Progress 100 is sent multiple times at the end
// this makes it so that we update the state only once.
if (progress === 100) now = now + 9999999999;
if (this.state.upload.lastUpdated + 500 < now){
this.setUploadState({
progress, totalBytes, totalBytesSent, lastUpdated: now
});
}
})
.on("addedfiles", files => {
this.setUploadState({
@ -164,7 +174,7 @@ class ProjectListItem extends React.Component {
})
.on("transformcompleted", (file, total) => {
if (this.dz._resizeMap) this.dz._resizeMap[file.name] = this.dz._taskInfo.resizeSize / Math.max(file.width, file.height);
this.setUploadState({resizedImages: total});
if (this.dz.options.resizeWidth) this.setUploadState({resizedImages: total});
})
.on("transformstart", (files) => {
if (this.dz.options.resizeWidth){

Wyświetl plik

@ -9,6 +9,9 @@ import AssetDownloadButtons from './AssetDownloadButtons';
import HistoryNav from '../classes/HistoryNav';
import PropTypes from 'prop-types';
import TaskPluginActionButtons from './TaskPluginActionButtons';
import PipelineSteps from '../classes/PipelineSteps';
import BasicTaskView from './BasicTaskView';
import Css from '../classes/Css';
class TaskListItem extends React.Component {
static propTypes = {
@ -32,7 +35,8 @@ class TaskListItem extends React.Component {
editing: false,
memoryError: false,
friendlyTaskError: "",
pluginActionButtons: []
pluginActionButtons: [],
view: "basic"
}
for (let k in props.data){
@ -44,9 +48,12 @@ class TaskListItem extends React.Component {
this.stopEditing = this.stopEditing.bind(this);
this.startEditing = this.startEditing.bind(this);
this.checkForCommonErrors = this.checkForCommonErrors.bind(this);
this.downloadTaskOutput = this.downloadTaskOutput.bind(this);
this.copyTaskOutput = this.copyTaskOutput.bind(this);
this.handleEditTaskSave = this.handleEditTaskSave.bind(this);
this.setView = this.setView.bind(this);
// Retrieve CSS values for status bar colors
this.backgroundSuccessColor = Css.getValue('theme-background-success', 'backgroundColor');
this.backgroundFailedColor = Css.getValue('theme-background-failed', 'backgroundColor');
}
shouldRefresh(){
@ -69,6 +76,12 @@ class TaskListItem extends React.Component {
}
}
setView(type){
return () => {
this.setState({view: type});
}
}
unloadTimer(){
if (this.processingTimeInterval){
clearInterval(this.processingTimeInterval);
@ -96,6 +109,7 @@ class TaskListItem extends React.Component {
if (oldStatus !== this.state.task.status){
if (this.state.task.status === statusCodes.RUNNING){
if (this.console) this.console.clear();
if (this.basicView) this.basicView.reset();
this.loadTimer(this.state.task.processing_time);
}else{
this.setState({time: this.state.task.processing_time});
@ -205,14 +219,6 @@ class TaskListItem extends React.Component {
};
}
downloadTaskOutput(){
this.console.downloadTxt("task_output.txt");
}
copyTaskOutput(){
this.console.copyTxt();
}
optionsToList(options){
if (!Array.isArray(options)) return "";
else if (options.length === 0) return "Default";
@ -269,35 +275,13 @@ class TaskListItem extends React.Component {
const { task } = this.state;
// Map rerun-from parameters to display items
const rfMap = {
"odm_meshing": {
label: "From Meshing",
icon: "fa fa-cube"
},
"mvs_texturing": {
label: "From Texturing",
icon: "fa fa-connectdevelop"
},
"odm_georeferencing": {
label: "From Georeferencing",
icon: "fa fa-globe"
},
"odm_dem": {
label: "From DEM",
icon: "fa fa-area-chart"
},
"odm_orthophoto": {
label: "From Orthophoto",
icon: "fa fa-map-o"
}
};
// (remove the first item so that 'dataset' is not displayed)
const rfMap = {};
PipelineSteps.get().slice(1).forEach(rf => rfMap[rf.action] = rf);
// Create onClick handlers
for (let rfParam in rfMap){
rfMap[rfParam].label = "From " + rfMap[rfParam].label;
rfMap[rfParam].onClick = this.genRestartAction(rfParam);
}
@ -372,7 +356,7 @@ class TaskListItem extends React.Component {
const name = task.name !== null ? task.name : `Task #${task.id}`;
let status = statusCodes.description(task.status);
if (status === "") status = "Uploading images";
if (status === "") status = "Uploading images to processing node";
if (!task.processing_node) status = "Waiting for a node...";
if (task.pending_action !== null) status = pendingActions.description(task.pending_action);
@ -468,24 +452,23 @@ class TaskListItem extends React.Component {
data-toggle="dropdown"><span className="caret"></span></button>,
<ul key="dropdown-menu" className="dropdown-menu">
{subItems.map(subItem => <li key={subItem.label}>
<a href="javascript:void(0);" onClick={subItem.onClick}><i className={subItem.icon}></i>{subItem.label}</a>
<a href="javascript:void(0);" onClick={subItem.onClick}><i className={subItem.icon + ' fa-fw '}></i>{subItem.label}</a>
</li>)}
</ul>]}
</div>);
})}
</div>);
expanded = (
<div className="expanded-panel">
<div className="row">
<div className="col-md-4 no-padding">
<div className="col-md-3 no-padding">
<div className="labels">
<strong>Created on: </strong> {(new Date(task.created_at)).toLocaleString()}<br/>
</div>
{status ? <div className="labels">
<strong>Status: </strong> {status}<br/>
</div>
: ""}
<div className="labels">
<strong>Processing Node: </strong> {task.processing_node_name || "-"} ({task.auto_processing_node ? "auto" : "manual"})<br/>
</div>
{Array.isArray(task.options) ?
<div className="labels">
<strong>Options: </strong> {this.optionsToList(task.options)}<br/>
@ -497,27 +480,39 @@ class TaskListItem extends React.Component {
<div className="task-warning"><i className="fa fa-warning"></i> <span>An orthophoto could not be generated. To generate one, make sure GPS information is embedded in the EXIF tags of your images, or use a Ground Control Points (GCP) file.</span></div> : ""}
</div>
<div className="col-md-8">
<Console
source={this.consoleOutputUrl}
refreshInterval={this.shouldRefresh() ? 3000 : undefined}
autoscroll={true}
height={200}
ref={domNode => this.console = domNode}
onAddLines={this.checkForCommonErrors}
/>
<div className="console-buttons">
<a href="javascript:void(0);" onClick={this.downloadTaskOutput} className="btn btn-sm btn-primary" title="Download Task Output">
<i className="fa fa-download"></i>
</a>
<a href="javascript:void(0);" onClick={this.copyTaskOutput} className="btn btn-sm btn-primary" title="Copy Task Output">
<i className="fa fa-clipboard"></i>
</a>
<div className="col-md-9">
<div className="switch-view text-right pull-right">
<i className="fa fa-list-ul"></i> <a href="javascript:void(0);" onClick={this.setView("basic")}
className={this.state.view === 'basic' ? "selected" : ""}>Simple</a>
|
<i className="fa fa-desktop"></i> <a href="javascript:void(0);" onClick={this.setView("console")}
className={this.state.view === 'console' ? "selected" : ""}>Console</a>
</div>
{this.state.view === 'console' ?
<Console
className="floatfix"
source={this.consoleOutputUrl}
refreshInterval={this.shouldRefresh() ? 3000 : undefined}
autoscroll={true}
height={200}
ref={domNode => this.console = domNode}
onAddLines={this.checkForCommonErrors}
showConsoleButtons={true}
maximumLines={500}
/> : ""}
{this.state.view === 'basic' ?
<BasicTaskView
source={this.consoleOutputUrl}
ref={domNode => this.basicView = domNode}
refreshInterval={this.shouldRefresh() ? 3000 : undefined}
onAddLines={this.checkForCommonErrors}
taskStatus={task.status}
/> : ""}
{showMemoryErrorWarning ?
<div className="task-warning"><i className="fa fa-support"></i> <span>It looks like your processing node ran out of memory. If you are using docker, make sure that your docker environment has <a href={memoryErrorLink} target="_blank">enough RAM allocated</a>. Alternatively, make sure you have enough physical RAM, reduce the number of images, make your images smaller, or reduce the max-concurrency parameter from the task's <a href="javascript:void(0);" onClick={this.startEditing}>options</a>.</span></div> : ""}
<div className="task-warning"><i className="fa fa-support"></i> <span>It looks like your processing node ran out of memory. If you are using docker, make sure that your docker environment has <a href={memoryErrorLink} target="_blank">enough RAM allocated</a>. Alternatively, make sure you have enough physical RAM, reduce the number of images, make your images smaller, or reduce the max-concurrency parameter from the task's <a href="javascript:void(0);" onClick={this.startEditing}>options</a>. You can also try to use a <a href="https://www.opendronemap.org/webodm/lightning/" target="_blank">cloud processing node</a>.</span></div> : ""}
{showTaskWarning ?
<div className="task-warning"><i className="fa fa-support"></i> <span dangerouslySetInnerHTML={{__html: this.state.friendlyTaskError}} /></div> : ""}
@ -529,7 +524,7 @@ class TaskListItem extends React.Component {
<li>Increase the <b>min-num-features</b> option, especially if your images have lots of vegetation</li>
</ul>
Still not working? Upload your images somewhere like <a href="https://www.dropbox.com/" target="_blank">Dropbox</a> or <a href="https://drive.google.com/drive/u/0/" target="_blank">Google Drive</a> and <a href="http://community.opendronemap.org/c/webodm" target="_blank">open a topic</a> on our community forum, making
sure to include a <a href="javascript:void(0);" onClick={this.downloadTaskOutput}>copy of your task's output</a> (the one you see above <i className="fa fa-arrow-up"></i>, click to <a href="javascript:void(0);" onClick={this.downloadTaskOutput}>download</a> it). Our awesome contributors will try to help you! <i className="fa fa-smile-o"></i>
sure to include a <a href="javascript:void(0);" onClick={this.setView("console")}>copy of your task's output</a>. Our awesome contributors will try to help you! <i className="fa fa-smile-o"></i>
</div>
</div>
: ""}
@ -557,8 +552,15 @@ class TaskListItem extends React.Component {
}
}
const getStatusLabel = (text, classes = "") => {
return (<div className={"status-label " + classes} title={text}>{text}</div>);
// @param type {String} one of: ['neutral', 'done', 'error']
const getStatusLabel = (text, type = 'neutral', progress = 100) => {
let color = 'rgba(255, 255, 255, 0.0)';
if (type === 'done') color = this.backgroundSuccessColor;
else if (type === 'error') color = this.backgroundFailedColor;
return (<div
className={"status-label theme-border-primary " + type}
style={{background: `linear-gradient(90deg, ${color} ${progress}%, rgba(255, 255, 255, 0) ${progress}%)`}}
title={text}>{text}</div>);
}
let statusLabel = "";
@ -566,13 +568,28 @@ class TaskListItem extends React.Component {
let showEditLink = false;
if (task.last_error){
statusLabel = getStatusLabel(task.last_error, "error");
statusLabel = getStatusLabel(task.last_error, 'error');
}else if (!task.processing_node){
statusLabel = getStatusLabel("Set a processing node");
statusIcon = "fa fa-hourglass-3";
showEditLink = true;
}else{
statusLabel = getStatusLabel(status, task.status == 40 ? "done" : "");
let progress = 100;
let type = 'done';
if (task.pending_action === pendingActions.RESIZE){
progress = task.resize_progress * 100;
}else if (task.status === null){
progress = task.upload_progress * 100;
}else if (task.status === statusCodes.RUNNING){
progress = task.running_progress * 100;
}else if (task.status === statusCodes.FAILED){
type = 'error';
}else if (task.status !== statusCodes.COMPLETED){
type = 'neutral';
}
statusLabel = getStatusLabel(status, type, progress);
}
return (

Wyświetl plik

@ -0,0 +1,10 @@
import React from 'react';
import { mount } from 'enzyme';
import BasicTaskView from '../BasicTaskView';
describe('<BasicTaskView />', () => {
it('renders without exploding', () => {
const wrapper = mount(<BasicTaskView source="http://localhost/output" taskStatus={40} />);
expect(wrapper.exists()).toBe(true);
})
});

Wyświetl plik

@ -0,0 +1,38 @@
.basic-task-view{
margin-bottom: 8px;
opacity: 0.7;
&.loaded{
opacity: 1;
}
.processing-step{
opacity: 0.7;
&.completed{
opacity: 1;
/* font-weight: bold; */
}
&.running{
opacity: 1;
animation: pulse ease-in-out 1.4833s infinite;
}
&.queued{
}
&.errored{
opacity: 1;
}
}
@keyframes pulse {
0% {
opacity: 0.7;
}
50% {
opacity: 1;
}
100% {
opacity: 0.7;
}
}
}

Wyświetl plik

@ -0,0 +1,9 @@
.console-buttons{
margin-left: 16px;
margin-bottom: 16px;
float: right;
text-align: right;
a{
margin-left: 4px;
}
}

Wyświetl plik

@ -41,6 +41,8 @@
padding: 4px;
width: 100%;
font-size: 90%;
border-style: solid;
border-width: 1px;
}
.clickable:hover{
@ -97,13 +99,25 @@
display: inline;
}
.console-buttons{
margin-left: 16px;
margin-bottom: 16px;
float: right;
text-align: right;
.switch-view{
margin-bottom: 8px;
font-size: 90%;
position: relative;
z-index: 99;
a{
margin-left: 4px;
margin-right: 8px;
&.selected{
font-weight: bold;
}
&:last-child{
margin-right: 0;
}
}
i{
margin-left: 8px;
}
}
}

Wyświetl plik

@ -14,7 +14,7 @@
{% if no_processingnodes %}
<h3>{% trans 'Welcome! ☺' %}</h3>
{% trans 'Add a Processing Node' as add_processing_node %}
{% with nodeodm_link='<a href="https://github.com/pierotofy/node-OpenDroneMap" target="_blank">node-OpenDroneMap</a>' api_link='<a href="https://github.com/pierotofy/node-OpenDroneMap/blob/master/docs/index.adoc" target="_blank">API</a>' %}
{% with nodeodm_link='<a href="https://github.com/OpenDroneMap/NodeODM" target="_blank">NodeODM</a>' api_link='<a href="https://github.com/OpenDroneMap/NodeODM/blob/master/docs/index.adoc" target="_blank">API</a>' %}
<p>
{% blocktrans %}
To get started, "{{ add_processing_node }}". A processing node is a computer running an instance of {{ nodeodm_link }} or some other software that implements this {{ api_link }}.

Wyświetl plik

@ -254,7 +254,7 @@
<ul class="nav nav-second-level">
{% for node in nodes %}
<li>
<a href="{% url 'processing_node' node.id %}"><span class="fa fa-laptop"></span> {{node}}</a>
<a href="{% url 'processing_node' node.id %}"><span class="fa fa-laptop {% if node.is_online %}theme-color-button-primary{% else %}theme-color-button-danger{% endif %}"></span> {{node}}</a>
</li>
{% endfor %}

Wyświetl plik

@ -23,6 +23,10 @@
<td>{% trans "Queue Count" %}</td>
<td>{{ processing_node.queue_count }}</td>
</tr>
<tr>
<td>{% trans "Max Images Limit" %}</td>
<td>{{ processing_node.max_images }}</td>
</tr>
<tr>
<td>{% trans "Last Refreshed" %}</td>
<td>{{ processing_node.last_refreshed|timesince }} {% trans 'ago' %} ({{ processing_node.last_refreshed|localtime }})</td> <!-- TODO: timezone? -->

Wyświetl plik

@ -350,6 +350,9 @@ class TestApi(BootTestCase):
# Should be set to false
self.assertFalse(res.data['online'])
# Verify max images field
self.assertTrue("max_images" in res.data)
# Cannot delete a processing node as normal user
res = client.delete('/api/processingnodes/{}/'.format(pnode.id))
self.assertTrue(res.status_code, status.HTTP_403_FORBIDDEN)

Wyświetl plik

@ -127,7 +127,6 @@ class TestApiTask(BootTransactionTestCase):
with Image.open(multiple_param_task.task_path("tiny_drone_image.jpg")) as im:
self.assertTrue(im.size == img1.size)
# Normal case with images[], GCP, name and processing node parameter and resize_to option
gcp = open("app/fixtures/gcp.txt", 'r')
res = client.post("/api/projects/{}/tasks/".format(project.id), {
@ -161,6 +160,13 @@ class TestApiTask(BootTransactionTestCase):
self.assertTrue(float(px) == 8.0) # Didn't change
self.assertTrue(float(py) == 8.0) # Didn't change
# Resize progress is 100%
resized_task.refresh_from_db()
self.assertEqual(resized_task.resize_progress, 1.0)
# Upload progress is 100%
self.assertEqual(resized_task.upload_progress, 1.0)
# Case with malformed GCP file option
with open("app/fixtures/gcp_malformed.txt", 'r') as malformed_gcp:
res = client.post("/api/projects/{}/tasks/".format(project.id), {
@ -181,7 +187,6 @@ class TestApiTask(BootTransactionTestCase):
image1.seek(0)
image2.seek(0)
# Cannot create a task with images[], name, but invalid processing node parameter
res = client.post("/api/projects/{}/tasks/".format(project.id), {
'images': [image1, image2],
@ -206,12 +211,18 @@ class TestApiTask(BootTransactionTestCase):
self.assertTrue('id' in res.data)
self.assertTrue(str(task.id) == res.data['id'])
# Progress is at 0%
self.assertEqual(task.running_progress, 0.0)
# Two images should have been uploaded
self.assertTrue(ImageUpload.objects.filter(task=task).count() == 2)
# Can_rerun_from should be an empty list
self.assertTrue(len(res.data['can_rerun_from']) == 0)
# processing_node_name should be null
self.assertTrue(res.data['processing_node_name'] is None)
# No processing node is set
self.assertTrue(task.processing_node is None)
@ -262,10 +273,33 @@ class TestApiTask(BootTransactionTestCase):
# (during tests this is sync)
# Processing should have started and a UUID is assigned
# Calling process pending tasks should finish the process
# and invoke the plugins completed signal
task.refresh_from_db()
self.assertTrue(task.status in [status_codes.RUNNING, status_codes.COMPLETED]) # Sometimes the task finishes and we can't test for RUNNING state
self.assertTrue(task.status in [status_codes.RUNNING, status_codes.COMPLETED]) # Sometimes this finishes before we get here
self.assertTrue(len(task.uuid) > 0)
with catch_signal(task_completed) as handler:
retry_count = 0
while task.status != status_codes.COMPLETED:
worker.tasks.process_pending_tasks()
time.sleep(DELAY)
task.refresh_from_db()
retry_count += 1
if retry_count > 10:
break
self.assertEqual(task.status, status_codes.COMPLETED)
# Progress is 100%
self.assertTrue(task.running_progress == 1.0)
handler.assert_any_call(
sender=Task,
task_id=task.id,
signal=task_completed,
)
# Processing node should have a "rerun_from" option
pnode_rerun_from_opts = list(filter(lambda d: 'name' in d and d['name'] == 'rerun-from', pnode.available_options))[0]
self.assertTrue(len(pnode_rerun_from_opts['domain']) > 0)
@ -277,20 +311,8 @@ class TestApiTask(BootTransactionTestCase):
self.assertTrue(res.status_code == status.HTTP_200_OK)
self.assertTrue(pnode_rerun_from_opts['domain'] == res.data['can_rerun_from'])
time.sleep(DELAY)
# Calling process pending tasks should finish the process
# and invoke the plugins completed signal
with catch_signal(task_completed) as handler:
worker.tasks.process_pending_tasks()
task.refresh_from_db()
self.assertTrue(task.status == status_codes.COMPLETED)
handler.assert_called_with(
sender=Task,
task_id=task.id,
signal=task_completed,
)
# processing_node_name should be the name of the pnode
self.assertEqual(res.data['processing_node_name'], str(pnode))
# Can download assets
for asset in list(task.ASSETS_MAP.keys()):

Wyświetl plik

@ -18,7 +18,7 @@ logger = logging.getLogger('app.logger')
def start_processing_node(*args):
current_dir = os.path.dirname(os.path.realpath(__file__))
node_odm = subprocess.Popen(['node', 'index.js', '--port', '11223', '--test'] + list(args), shell=False,
cwd=os.path.join(current_dir, "..", "..", "nodeodm", "external", "node-OpenDroneMap"))
cwd=os.path.join(current_dir, "..", "..", "nodeodm", "external", "NodeODM"))
time.sleep(2) # Wait for the server to launch
return node_odm

Wyświetl plik

@ -4,3 +4,6 @@ services:
entrypoint: /bin/bash -c "chmod +x /webodm/*.sh && /bin/bash -c \"/webodm/wait-for-postgres.sh db /webodm/wait-for-it.sh -t 0 broker:6379 -- /webodm/start.sh --create-default-pnode --setup-devenv\""
volumes:
- .:/webodm
worker:
volumes:
- .:/webodm

Wyświetl plik

@ -9,7 +9,7 @@ services:
depends_on:
- node-odm-1
node-odm-1:
image: opendronemap/node-opendronemap
image: opendronemap/nodeodm
container_name: node-odm-1
ports:
- "3000"

Wyświetl plik

@ -1,7 +1,9 @@
"""
An interface to node-OpenDroneMap's API
https://github.com/pierotofy/node-OpenDroneMap/blob/master/docs/index.adoc
An interface to NodeODM's API
https://github.com/pierotofy/NodeODM/blob/master/docs/index.adoc
"""
from requests.packages.urllib3.fields import RequestField
from requests_toolbelt.multipart import encoder
import requests
import mimetypes
import json
@ -9,6 +11,47 @@ import os
from urllib.parse import urlunparse, urlencode
from app.testwatch import TestWatch
# Extends class to support multipart form data
# fields with the same name
# https://github.com/requests/toolbelt/issues/225
class MultipartEncoder(encoder.MultipartEncoder):
"""Multiple files with the same name support, i.e. files[]"""
def _iter_fields(self):
_fields = self.fields
if hasattr(self.fields, 'items'):
_fields = list(self.fields.items())
for k, v in _fields:
for field in self._iter_field(k, v):
yield field
@classmethod
def _iter_field(cls, field_name, field):
file_name = None
file_type = None
file_headers = None
if field and isinstance(field, (list, tuple)):
if all([isinstance(f, (list, tuple)) for f in field]):
for f in field:
yield next(cls._iter_field(field_name, f))
else:
raise StopIteration()
if len(field) == 2:
file_name, file_pointer = field
elif len(field) == 3:
file_name, file_pointer, file_type = field
else:
file_name, file_pointer, file_type, file_headers = field
else:
file_pointer = field
field = RequestField(name=field_name,
data=file_pointer,
filename=file_name,
headers=file_headers)
field.make_multipart(content_type=file_type)
yield field
class ApiClient:
def __init__(self, host, port, token = "", timeout=30):
self.host = host
@ -17,13 +60,13 @@ class ApiClient:
self.timeout = timeout
def url(self, url, query = {}):
netloc = self.host if self.port == 80 else "{}:{}".format(self.host, self.port)
netloc = self.host if (self.port == 80 or self.port == 443) else "{}:{}".format(self.host, self.port)
proto = 'https' if self.port == 443 else 'http'
if len(self.token) > 0:
query['token'] = self.token
# TODO: https support
return urlunparse(('http', netloc, url, '', urlencode(query), ''))
return urlunparse((proto, netloc, url, '', urlencode(query), ''))
def info(self):
return requests.get(self.url('/info'), timeout=self.timeout).json()
@ -56,12 +99,13 @@ class ApiClient:
else:
return res
def new_task(self, images, name=None, options=[]):
def new_task(self, images, name=None, options=[], progress_callback=None):
"""
Starts processing of a new task
:param images: list of path images
:param name: name of the task
:param options: options to be used for processing ([{'name': optionName, 'value': optionValue}, ...])
:param progress_callback: optional callback invoked during the upload images process to be used to report status.
:return: UUID or error
"""
@ -72,9 +116,24 @@ class ApiClient:
with open(path, 'rb') as f:
return f.read()
files = [('images',
(os.path.basename(image), read_file(image), (mimetypes.guess_type(image)[0] or "image/jpg"))
) for image in images]
fields = {
'name': name,
'options': json.dumps(options),
'images': [(os.path.basename(image), read_file(image), (mimetypes.guess_type(image)[0] or "image/jpg")) for image in images]
}
def create_callback(mpe):
total_bytes = mpe.len
def callback(monitor):
if progress_callback is not None and total_bytes > 0:
progress_callback(monitor.bytes_read / total_bytes)
return callback
e = MultipartEncoder(fields=fields)
m = encoder.MultipartEncoderMonitor(e, create_callback(e))
return requests.post(self.url("/task/new"),
files=files,
data={'name': name, 'options': json.dumps(options)}).json()
data=m,
headers={'Content-Type': m.content_type}).json()

1
nodeodm/external/NodeODM vendored 160000

@ -0,0 +1 @@
Subproject commit d0e1e1424e90c21fee401ce6e5a7dd1de7b32112

@ -1 +0,0 @@
Subproject commit 1c9e149a7c32da438f7844b8cfb81bec74a050d4

Wyświetl plik

@ -0,0 +1,18 @@
# Generated by Django 2.0.3 on 2018-12-04 17:04
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('nodeodm', '0003_auto_20180625_1230'),
]
operations = [
migrations.AddField(
model_name='processingnode',
name='max_images',
field=models.PositiveIntegerField(blank=True, help_text='Maximum number of images accepted by this node.', null=True),
),
]

Wyświetl plik

@ -42,6 +42,7 @@ class ProcessingNode(models.Model):
queue_count = models.PositiveIntegerField(default=0, help_text="Number of tasks currently being processed by this node (as reported by the node itself)")
available_options = fields.JSONField(default=dict(), help_text="Description of the options that can be used for processing")
token = models.CharField(max_length=1024, blank=True, default="", help_text="Token to use for authentication. If the node doesn't have authentication, you can leave this field blank.")
max_images = models.PositiveIntegerField(help_text="Maximum number of images accepted by this node.", blank=True, null=True)
def __str__(self):
return '{}:{}'.format(self.hostname, self.port)
@ -70,9 +71,15 @@ class ProcessingNode(models.Model):
api_client = self.api_client(timeout=5)
try:
info = api_client.info()
if 'error' in info:
return False
self.api_version = info['version']
self.queue_count = info['taskQueueCount']
if 'maxImages' in info:
self.max_images = info['maxImages']
options = api_client.options()
self.available_options = options
self.last_refreshed = timezone.now()
@ -92,7 +99,7 @@ class ProcessingNode(models.Model):
return json.dumps(self.available_options, **kwargs)
@api
def process_new_task(self, images, name=None, options=[]):
def process_new_task(self, images, name=None, options=[], progress_callback=None):
"""
Sends a set of images (and optional GCP file) via the API
to start processing.
@ -100,6 +107,7 @@ class ProcessingNode(models.Model):
:param images: list of path images
:param name: name of the task
:param options: options to be used for processing ([{'name': optionName, 'value': optionValue}, ...])
:param progress_callback: optional callback invoked during the upload images process to be used to report status.
:returns UUID of the newly created task
"""
@ -107,7 +115,7 @@ class ProcessingNode(models.Model):
api_client = self.api_client()
try:
result = api_client.new_task(images, name, options)
result = api_client.new_task(images, name, options, progress_callback)
except requests.exceptions.ConnectionError as e:
raise ProcessingError(e)
@ -145,7 +153,7 @@ class ProcessingNode(models.Model):
if isinstance(result, dict) and 'error' in result:
raise ProcessingError(result['error'])
elif isinstance(result, list):
return "\n".join(result)
return result
else:
raise ProcessingError("Unknown response for console output: {}".format(result))
@ -189,7 +197,7 @@ class ProcessingNode(models.Model):
def handle_generic_post_response(result):
"""
Handles a POST response that has either a "success" flag, or an error message.
This is a common response in node-OpenDroneMap POST calls.
This is a common response in NodeODM POST calls.
:param result: result of API call
:return: True on success, raises ProcessingException otherwise
"""

Wyświetl plik

@ -21,7 +21,7 @@ class TestClientApi(TestCase):
@classmethod
def setUpClass(cls):
super(TestClientApi, cls).setUpClass()
cls.node_odm = subprocess.Popen(['node', 'index.js', '--port', '11223', '--test'], shell=False, cwd=path.join(current_dir, "external", "node-OpenDroneMap"))
cls.node_odm = subprocess.Popen(['node', 'index.js', '--port', '11223', '--test'], shell=False, cwd=path.join(current_dir, "external", "NodeODM"))
time.sleep(2) # Wait for the server to launch
@ -45,6 +45,7 @@ class TestClientApi(TestCase):
info = self.api_client.info()
self.assertTrue(isinstance(info['version'], six.string_types), "Found version string")
self.assertTrue(isinstance(info['taskQueueCount'], int), "Found task queue count")
self.assertTrue(info['maxImages'] is None, "Found task max images")
def test_options(self):
options = self.api_client.options()
@ -58,9 +59,10 @@ class TestClientApi(TestCase):
self.assertTrue(online_node.api_version == "", "API version is not set")
self.assertTrue(online_node.update_node_info(), "Could update info")
self.assertTrue(online_node.last_refreshed != None, "Last refreshed is set")
self.assertTrue(online_node.last_refreshed is not None, "Last refreshed is set")
self.assertTrue(len(online_node.available_options) > 0, "Available options are set")
self.assertTrue(online_node.api_version != "", "API version is set")
self.assertTrue(online_node.max_images is None, "No max images limit is set")
self.assertTrue(isinstance(online_node.get_available_options_json(), six.string_types), "Available options json works")
self.assertTrue(isinstance(online_node.get_available_options_json(pretty=True), six.string_types), "Available options json works with pretty")
@ -121,7 +123,7 @@ class TestClientApi(TestCase):
# task_output
self.assertTrue(isinstance(api.task_output(uuid, 0), list))
self.assertTrue(isinstance(online_node.get_task_console_output(uuid, 0), str))
self.assertTrue(isinstance(online_node.get_task_console_output(uuid, 0), list))
self.assertRaises(ProcessingError, online_node.get_task_console_output, "wrong-uuid", 0)
@ -157,6 +159,10 @@ class TestClientApi(TestCase):
# Task has been deleted
self.assertRaises(ProcessingError, online_node.get_task_info, uuid)
# Test URL building for HTTPS
sslApi = ApiClient("localhost", 443, 'abc')
self.assertEqual(sslApi.url('/info'), 'https://localhost/info?token=abc')
def test_find_best_available_node_and_is_online(self):
# Fixtures are all offline
self.assertTrue(ProcessingNode.find_best_available_node() is None)
@ -191,7 +197,7 @@ class TestClientApi(TestCase):
def test_token_auth(self):
node_odm = subprocess.Popen(
['node', 'index.js', '--port', '11224', '--token', 'test_token', '--test'], shell=False,
cwd=path.join(current_dir, "external", "node-OpenDroneMap"))
cwd=path.join(current_dir, "external", "NodeODM"))
time.sleep(2)
def wait_for_status(api, uuid, status, num_retries=10, error_description="Failed to wait for status"):
@ -215,10 +221,10 @@ class TestClientApi(TestCase):
self.assertTrue(online_node.update_node_info(), "Could update info")
# Can always call info(), options() (even without valid tokens)
# Cannot call info(), options() without tokens
api.token = "invalid"
self.assertTrue(type(api.info()['version']) == str)
self.assertTrue(len(api.options()) > 0)
self.assertTrue(type(api.info()['error']) == str)
self.assertTrue(type(api.options()['error']) == str)
# Cannot call new_task() without token
import glob

Wyświetl plik

@ -1,6 +1,6 @@
{
"name": "WebODM",
"version": "0.6.2",
"version": "0.7.0",
"description": "Open Source Drone Image Processing",
"main": "index.js",
"scripts": {

Wyświetl plik

@ -41,6 +41,7 @@ pyparsing==2.1.10
pytz==2018.3
rcssmin==1.0.6
redis==2.10.6
requests-toolbelt==0.8.0
requests==2.20.0
rfc3987==1.3.7
rjsmin==1.0.12

Wyświetl plik

@ -51,7 +51,7 @@ Directories of interest are listed as follow:
Directory | Description
--------- | -----------
`/app` | Main application, includes the UI components, API, tests and backend logic.
`/nodeodm`| Application that bridges the communication between WebODM and [node-OpenDroneMap](https://github.com/OpenDroneMap/node-OpenDroneMap). Includes its own unit tests and models.
`/nodeodm`| Application that bridges the communication between WebODM and [NodeODM](https://github.com/OpenDroneMap/NodeODM). Includes its own unit tests and models.
`/webodm` | Django's main project directory. Setting files are here.
### Frontend

Wyświetl plik

@ -11,6 +11,7 @@
"api_version": "1.0.1",
"last_refreshed": "2017-03-01T21:14:49.918276Z",
"queue_count": 0,
"max_images": null,
"available_options": [
{
"help": "Oct-tree depth at which the Laplacian equation is solved in the surface reconstruction step. Increasing this value increases computation times slightly but helps reduce memory usage. Default: 9",
@ -23,7 +24,7 @@
```
Processing nodes are associated with zero or more tasks and
take care of processing input images. Processing nodes are computers or virtual machines running [node-OpenDroneMap](https://github.com/OpenDroneMap/node-OpenDroneMap/) or any other API compatible with it.
take care of processing input images. Processing nodes are computers or virtual machines running [NodeODM](https://github.com/OpenDroneMap/NodeODM) or any other API compatible with it.
Field | Type | Description
----- | ---- | -----------
@ -31,9 +32,10 @@ id | int | Unique Identifier
online | bool | Whether the processing node could be reached in the last 5 minutes
hostname | string | Hostname/IP address
port | int | Port
api_version | string | Version of node-OpenDroneMap currently running
api_version | string | Version of NodeODM currently running
last_refreshed | string | Date and time this node was last seen online. This value is typically refreshed every 15-30 seconds and is used to decide whether a node is offline or not
queue_count | int | Number of [Task](#task) items currently being processed/queued on this node.
max_images | int | Optional maximum number of images this processing node can accept. null indicates no limit.
available_options | JSON[] | JSON-encoded list of options that this node is capable of handling. See [Available Options](#available-options) for more information
@ -45,7 +47,7 @@ help | Description of the option
name | Name that identifies the option. This is the value you pass in the `name` key/value pair when creating a set of options for a new [Task](#task)
type | Possible values are `int`, `float`, `string`, `bool`
value | Default value if the option is not specified
domain | Restriction of the range of values that this option allows. Examples are `float`, `negative integer`, `percent`, `float: 0 <= x <= 10`, etc. for all possible values, check [node-OpenDroneMap's odmOptions.js code](https://github.com/OpenDroneMap/node-OpenDroneMap/blob/master/libs/odmOptions.js#L135)
domain | Restriction of the range of values that this option allows. Examples are `float`, `negative integer`, `percent`, `float: 0 <= x <= 10`, etc. for all possible values, check [NodeODM's odmOptions.js code](https://github.com/OpenDroneMap/NodeODM/blob/master/libs/odmOptions.js#L135)
### Add a processing node
@ -83,6 +85,7 @@ hostname | | "" | Filter by hostname
port | | "" | Filter by port
api_version | | "" | Filter by API version
queue_count | | "" | Filter by queue count
max_images | | "" | Filter by max images
ordering | | "" | Ordering field to sort results by
has_available_options | | "" | Return only processing nodes that have a valid set of processing options (check that the `available_options` field is populated). Either `true` or `false`.
@ -128,5 +131,5 @@ Display the common options available among all online processing nodes. This is
Use this list of options to check whether a particular option is supported by all online processing nodes. If you use the automatic processing node assignment feature for processing tasks, this is the list you want to display to the user for choosing the options to use during processing.
<aside class="notice">While WebODM is capable of handling processing nodes running different versions of node-OpenDroneMap, we don't recommend doing so. When all processing nodes use the same node-OpenDroneMap version, the output of this API call will be identical to the <b>available_options</b> field of any node.</aside>
<aside class="notice">While WebODM is capable of handling processing nodes running different versions of NodeODM, we don't recommend doing so. When all processing nodes use the same NodeODM version, the output of this API call will be identical to the <b>available_options</b> field of any node.</aside>

Wyświetl plik

@ -7,6 +7,7 @@
"id": 134,
"project": 27,
"processing_node": 10,
"processing_node_name": "localhost:3000",
"images_count": 48,
"can_rerun_from": [],
"available_assets": [
@ -30,9 +31,11 @@
"value": true
}
],
"ground_control_points": null,
"created_at": "2017-02-18T18:01:55.402551Z",
"pending_action": null
"pending_action": null,
"upload_progress": 1.0,
"resize_progress": 0.0,
"running_progress": 1.0
}
```
@ -43,6 +46,7 @@ Field | Type | Description
id | int | Unique identifier
project | int | [Project](#project) ID the task belongs to
processing_node | int | The ID of the [Processing Node](#processing-node) this task has been assigned to, or `null` if no [Processing Node](#processing-node) has been assigned.
processing_node_name | string | The name of the processing node below, or `null` if no [Processing Node](#processing-node) has been assigned.
images_count | int | Number of images
can_rerun_from | string[] | List of possible "rerun-from" options that this task could restart from, given its currently assigned processing node. If this is an empty list, the task can only be restarted from the start of the pipeline.
available_assets | string[] | List of [assets](#download-assets) available for download
@ -53,9 +57,12 @@ auto_processing_node | boolean | Whether WebODM should automatically assign the
status | int | One of [Status Codes](#status-codes), or `null` if no status is available.
last_error | string | The last error message reported by a [Processing Node](#processing-node) in case of processing failure.
options | JSON[] | JSON-encoded list of name/value pairs, where each pair represents a command line option to be passed to a [Processing Node](#processing-node).
ground_control_points | string | Currently unused. See [#37](https://github.com/OpenDroneMap/WebODM/issues/37)
created_at | string | Creation date and time
created_at | string | Creation date and time.
pending_action | int | One of [Pending Actions](#pending-actions), or `null` if no pending action is set.
upload_progress | float | Value between 0 and 1 indicating the upload progress of this task's files to the processing node.
resize_progress | float | Value between 0 and 1 indicating the resize progress of this task's images.
running_progress | float | Value between 0 and 1 indicating the running progress (estimated) of this task.
<aside class="notice">Tasks inherit the permission settings from the <a href="#project">Project</a> they belong to.</aside>
@ -91,6 +98,7 @@ Parameters are the same as above.
"id": 6,
"project": 2,
"processing_node": 2,
"processing_node_name": "localhost:3000",
"images_count": 89,
"uuid": "2e8b687d-c269-4e2f-91b3-5a2cd51b5321",
"name": "Test name",
@ -99,9 +107,11 @@ Parameters are the same as above.
"status": 40,
"last_error": null,
"options": [],
"ground_control_points": null,
"created_at": "2016-12-08T13:32:28.139474Z",
"pending_action": null
"pending_action": null,
"upload_progress": 1.0,
"resize_progress": 0.0,
"running_progress": 1.0
}
]
```

Wyświetl plik

@ -6,8 +6,8 @@ language_tabs:
toc_footers:
- <a href='https://github.com/OpenDroneMap/WebODM'>WebODM on GitHub</a>
- <a href='https://github.com/OpenDroneMap/OpenDroneMap'>OpenDroneMap on GitHub</a>
- <a href='https://github.com/OpenDroneMap/node-OpenDroneMap'>node-OpenDroneMap on GitHub</a>
- <a href='https://github.com/OpenDroneMap/ODM'>ODM on GitHub</a>
- <a href='https://github.com/OpenDroneMap/NodeODM'>NodeODM on GitHub</a>
search: true

Wyświetl plik

@ -43,10 +43,13 @@ if [ "$1" = "--setup-devenv" ] || [ "$2" = "--setup-devenv" ]; then
echo Setup npm dependencies...
npm install
cd nodeodm/external/node-OpenDroneMap
cd nodeodm/external/NodeODM
npm install
cd /webodm
echo Setup pip requirements...
pip install -r requirements.txt
echo Setup webpack watch...
webpack --watch &
fi

Wyświetl plik

@ -248,7 +248,7 @@ rebuild(){
run "docker-compose down --remove-orphans"
plugin_cleanup
run "rm -fr node_modules/ || sudo rm -fr node_modules/"
run "rm -fr nodeodm/external/node-OpenDroneMap || sudo rm -fr nodeodm/external/node-OpenDroneMap"
run "rm -fr nodeodm/external/NodeODM || sudo rm -fr nodeodm/external/NodeODM"
run "docker-compose -f docker-compose.yml -f docker-compose.build.yml build --no-cache"
#run "docker images --no-trunc -aqf \"dangling=true\" | xargs docker rmi"
echo -e "\033[1mDone!\033[0m You can now start WebODM by running $0 start"
@ -371,7 +371,7 @@ elif [[ $1 = "rebuild" ]]; then
elif [[ $1 = "update" ]]; then
echo "Updating WebODM..."
run "git pull origin master"
run "docker pull opendronemap/node-opendronemap"
run "docker pull opendronemap/nodeodm"
run "docker pull opendronemap/webodm_db"
run "docker pull opendronemap/webodm_webapp"
run "docker-compose down --remove-orphans"