Merge pull request #510 from dakotabenjamin/global-config-file

Global settings file, Versioning, GDAL creation options

Former-commit-id: 7b440295db

.gitignore

@@ -17,5 +17,6 @@ LAStools.zip
 pcl.tar.gz
 ceres-solver.tar.gz
 *.pyc
-pcl.tar.gz
 opencv.zip
+settings.yaml
+docker.settings.yaml

Dockerfile

@@ -19,6 +19,7 @@ COPY run.py /code/run.py
 COPY /scripts/ /code/scripts/
 COPY /SuperBuild/cmake/ /code/SuperBuild/cmake/
 COPY /SuperBuild/CMakeLists.txt /code/SuperBuild/CMakeLists.txt
+COPY docker.settings.yaml /code/settings.yaml
 COPY /tests/ /code/tests/
 
 # Update submodules
@@ -29,4 +30,4 @@ RUN cd SuperBuild && mkdir build && cd build && cmake .. && make -j$(nproc) \
     && cd ../.. && mkdir build && cd build && cmake .. && make -j$(nproc)
 
 # Entry point
-ENTRYPOINT ["python", "/code/run.py", "--project-path", "/code/"]
+ENTRYPOINT ["python", "/code/run.py", "code"]

README.md

@@ -33,8 +33,9 @@ Current version: 0.2 (this software is in beta)
 
 1. Extract and enter the OpenDroneMap directory
 2. Run `bash configure.sh`
-3. Download and extract a sample dataset [here](https://github.com/OpenDroneMap/odm_data_aukerman/archive/master.zip)
-4. Run `./run.sh --project-path <PATH>`, replacing `<PATH>` with the dataset path. 
+4. Copy the default settings file and edit it: `cp default.settings.yaml settings.yaml`. Set the `project-path` value to an empty directory (you will place sub-directories containing individual projects inside). You can add many options to this file, [see here](https://github.com/OpenDroneMap/OpenDroneMap/wiki/Run-Time-Parameters)
+3. Download a sample dataset from [here](https://github.com/OpenDroneMap/odm_data_aukerman/archive/master.zip) (about 550MB) and extract it as a subdirectory in your project directory.
+4. Run `./run.sh odm_data_aukerman` 
 5. Enter dataset directory to view results: 
   - orthophoto: odm_orthophoto/odm_orthophoto.tif
   - textured mesh model: odm_texturing/odm_textured_model_geo.obj
@@ -65,13 +66,24 @@ Note that using `run.sh` sets these temporarily in the shell.
     
 ### Run OpenDroneMap
 
-First you need a set of images, taken from a drone or otherwise. Example data can be cloned from https://github.com/OpenDroneMap/odm_data
+First you need a set of images, taken from a drone or otherwise. Example data can be obtained from https://github.com/OpenDroneMap/odm_data
+
+Next, you need to copy over the settings file `default.settings.yaml` and edit it. The only setting you must edit is the `project-path` key. Set this to an empty directory within projects will be saved. There are many options for tuning your project. See the [wiki](https://github.com/OpenDroneMap/OpenDroneMap/wiki/Run-Time-Parameters) or run `python run.py -h`
+
 
 Then run:
 
-    python run.py --project-path /path/to/project -i /path/to/images
+    python run.py -i /path/to/images project-name
 
-The images will be copied over to the project path so you only need to specify the `-i /path/` once. There are many options for tuning your project. See the [wiki](https://github.com/OpenDroneMap/OpenDroneMap/wiki/Run-Time-Parameters) or run `python run.py -h`
+The images will be copied over to the project path so you only need to specify the `-i /path/` once. You can also override any variable from settings.yaml here using the command line arguments. If you want to rerun the whole thing, run
+
+    python run.py --rerun-all project-name
+    
+or
+
+    python run.py --rerun-from odm_meshing project-name
+
+The options for rerunning are: 'resize', 'opensfm', 'slam', 'cmvs', 'pmvs', 'odm_meshing', 'mvs_texturing', 'odm_georeferencing', 'odm_orthophoto'
 
 ### View Results
 
@@ -85,10 +97,8 @@ When the process finishes, the results will be organized as follows:
         |-- ...
     |-- opensfm/
         |-- see mapillary/opensfm repository for more info
-    |-- pmvs/
-        |-- recon0/
-            |-- models/
-                |-- option-0000.ply         # Dense point cloud (not georeferenced)
+        |-- depthmaps/
+            |-- merged.ply                  # Dense Point cloud (not georeferenced)
     |-- odm_meshing/
         |-- odm_mesh.ply                    # A 3D mesh
         |-- odm_meshing_log.txt             # Output of the meshing task. May point out errors.
@@ -102,13 +112,13 @@ When the process finishes, the results will be organized as follows:
         |-- odm_georeferenced_model.csv     # XYZ format point cloud
         |-- odm_georeferencing_log.txt      # Georeferencing log
         |-- odm_georeferencing_utm_log.txt  # Log for the extract_utm portion
-    |-- odm_georeferencing/
+    |-- odm_orthophoto/
         |-- odm_orthophoto.png              # Orthophoto image (no coordinates)
         |-- odm_orthophoto.tif              # Orthophoto GeoTiff
         |-- odm_orthophoto_log.txt          # Log file
         |-- gdal_translate_log.txt          # Log for georeferencing the png file
 
-Any file ending in .obj or .ply can be opened and viewed in [MeshLab](http://meshlab.sourceforge.net/) or similar software. That includes `pmvs/recon0/models/option-000.ply`, `odm_meshing/odm_mesh.ply`, `odm_texturing/odm_textured_model[_geo].obj`, or `odm_georeferencing/odm_georeferenced_model.ply`. Below is an example textured mesh:
+Any file ending in .obj or .ply can be opened and viewed in [MeshLab](http://meshlab.sourceforge.net/) or similar software. That includes `opensfm/depthmaps/merged.ply`, `odm_meshing/odm_mesh.ply`, `odm_texturing/odm_textured_model[_geo].obj`, or `odm_georeferencing/odm_georeferenced_model.ply`. Below is an example textured mesh:
 
 ![](https://raw.githubusercontent.com/alexhagiopol/OpenDroneMap/feature-better-docker/toledo_dataset_example_mesh.jpg)
 
@@ -131,7 +141,6 @@ instructions through "Create a Docker group". Once Docker is installed, the fast
 If you want to build your own Docker image from sources, type:
 
     docker build -t packages -f packages.Dockerfile .
-
     docker build -t my_odm_image .
     docker run -it --rm -v $(pwd)/images:/code/images -v $(pwd)/odm_orthophoto:/code/odm_orthophoto -v $(pwd)/odm_texturing:/code/odm_texturing my_odm_image
 
@@ -141,7 +150,7 @@ If you want to view other results outside the Docker image simply add which dire
 established above. For example, if you're interested in the dense cloud results generated by PMVS and in the orthophoto,
 simply use the following `docker run` command after building the image:
 
-    docker run -it --rm -v $(pwd)/images:/code/images -v $(pwd)/pmvs:/code/pmvs -v $(pwd)/odm_orthophoto:/code/odm_orthophoto my_odm_image
+    docker run -it --rm -v $(pwd)/images:/code/images -v $(pwd)/odm_georeferencing:/code/odm_georeferencing -v $(pwd)/odm_orthophoto:/code/odm_orthophoto my_odm_image
 
 If you want to get all intermediate outputs, run the following command:
 
@@ -151,6 +160,11 @@ To pass in custom parameters to the run.py script, simply pass it as arguments t
 
     docker run -it --rm -v $(pwd)/images:/code/images v $(pwd)/odm_orthophoto:/code/odm_orthophoto -v $(pwd)/odm_texturing:/code/odm_texturing opendronemap/opendronemap --resize-to 1800 --force-ccd 6.16
 
+If you want to pass in custom parameters using the settings.yaml file, you can pass it as a -v volume binding:
+
+    docker run -it --rm -v $(pwd)/images:/code/images v $(pwd)/odm_orthophoto:/code/odm_orthophoto -v $(pwd)/odm_texturing:/code/odm_texturing -v $(pwd)/settings.yaml:/code/settings.yaml opendronemap/opendronemap
+
+
 ## User Interface
 
 A web interface and API to OpenDroneMap is currently under active development in the [WebODM](https://github.com/OpenDroneMap/WebODM) repository.

VERSION

@@ -0,0 +1 @@
+0.3-development

configure.sh

@@ -67,7 +67,8 @@ sudo apt-get install -y -qq python-networkx \
 sudo pip install -U PyYAML \
                     exifread \
                     gpxpy \
-                    xmltodict
+                    xmltodict \
+                    appsettings
 
 echo "Installing Ecto Dependencies"
 sudo pip install -U catkin-pkg

docker.settings.yaml

@@ -0,0 +1,49 @@
+---
+# A list of global configuration variables
+# Uncomment lines as needed to edit default settings.
+# Note this only works for settings with default values. Some commands like --rerun <module>
+# or --force-ccd n will have to be set in the command line (if you need to)
+
+# This line is really important to set up properly
+project_path: '/' #DO NOT CHANGE THIS OR DOCKER WILL NOT WORK. It should be '/'
+
+# The rest of the settings will default to the values set unless you uncomment and change them
+#resize_to: 2400
+#start_with: 'resize'
+#end_with: 'odm_orthophoto'
+#rerun_all: False
+#zip_results: False
+#verbose: False
+#time: False
+#opensfm_processes: 4 # by default this is set to $(nproc)
+#min_num_features: 4000
+#matcher_threshold: 2.0
+#matcher_ratio: 0.6
+#matcher_neighbors: 8
+#matcher_distance: 0
+#use_pmvs: False # The cmvs/pmvs settings only matter if 'Enabled' is set to True
+#cmvs_maximages: 500
+#pmvs_level: 1
+#pmvs_csize: 2
+#pmvs_threshold: 0.7
+#pmvs_wsize: 7
+#pmvs_min_images: 3
+#pmvs_num_cores: 4 # by default this is set to $(nproc)
+#mesh_size: 100000
+#mesh_octree_depth: 9
+#mesh_samples: 1.0
+#mesh_solver_divide: 9
+#texturing_data_term: 'gmi'
+#texturing_outlier_removal_type: 'gauss_clamping'
+#texturing_skip_visibility_test: False
+#texturing_skip_global_seam_leveling: False
+#texturing_skip_local_seam_leveling: False
+#texturing_skip_hole_filling: False
+#texturing_keep_unseen_faces: False
+#texturing_tone_mapping: 'none'
+#gcp: !!null  # YAML tag for None
+#use_exif: False # Set to True if you have a GCP file (it auto-detects) and want to use EXIF
+#orthophoto_resolution: 20.0 # Pixels/meter
+#orthophoto_target_srs: !!null # Currently does nothing
+#orthophoto_no_tiled: False
+#orthophoto_compression: DEFLATE # Options are [JPEG, LZW, PACKBITS, DEFLATE, LZMA, NONE] Don't change unless you know what you are doing

opendm/config.py

@@ -1,28 +1,51 @@
 import argparse
-import context
+from opendm import context
+from opendm import io
+from opendm import log
+from yaml import safe_load
+from appsettings import SettingsParser
+
+import sys
 
 # parse arguments
 processopts = ['resize', 'opensfm', 'slam', 'cmvs', 'pmvs',
                'odm_meshing', 'mvs_texturing', 'odm_georeferencing',
                'odm_orthophoto']
 
+with open(io.join_paths(context.root_path, 'VERSION')) as version_file:
+    __version__ = version_file.read().strip()
+
+
+def alphanumeric_string(string):
+    import re
+    if re.match('^[a-zA-Z0-9_-]+$', string) is None:
+        msg = '{0} is not a valid name. Must use alphanumeric characters.'.format(string)
+        raise argparse.ArgumentTypeError(msg)
+    return string
+
 
 class RerunFrom(argparse.Action):
     def __call__(self, parser, namespace, values, option_string=None):
         setattr(namespace, self.dest, processopts[processopts.index(values):])
 
 
-parser = argparse.ArgumentParser(description='OpenDroneMap')
-
+parser = SettingsParser(description='OpenDroneMap',
+                        usage='%(prog)s [options] <project name>',
+                        yaml_file=open(context.settings_path))
 
 def config():
     parser.add_argument('--images', '-i',
-                        metavar='<string>',
+                        metavar='<path>',
                         help='Path to input images'),
 
     parser.add_argument('--project-path',
-                        metavar='<string>',
-                        help='Path to the project to process')
+                        metavar='<path>',
+                        help='Path to the project folder')
+
+    parser.add_argument('name',
+                        metavar='<project name>',
+                        type=alphanumeric_string,
+                        help='Name of Project (i.e subdirectory of projects folder)')
 
     parser.add_argument('--resize-to',  # currently doesn't support 'orig'
                         metavar='<integer>',
@@ -119,7 +142,8 @@ def config():
                         default=0,
                         type=int,
                         help='Distance threshold in meters to find pre-matching '
-                             'images based on GPS exif data. Set to 0 to skip '
+                             'images based on GPS exif data. Set both '
+                             'matcher-neighbors and this to 0 to skip '
                              'pre-matching. Default: %(default)s')
 
     parser.add_argument('--opensfm-processes',
@@ -132,8 +156,7 @@ def config():
     parser.add_argument('--use-pmvs',
                         action='store_true',
                         default=False,
-                        help='Use OpenSfM to compute the point cloud instead '
-                             'of PMVS')
+                        help='Use pmvs to compute point cloud alternatively')
 
     parser.add_argument('--cmvs-maxImages',
                         metavar='<integer>',
@@ -152,7 +175,7 @@ def config():
                               'more pmvs documentation. Default: %(default)s'))
 
     parser.add_argument('--pmvs-csize',
-                        metavar='< positive integer>',
+                        metavar='<positive integer>',
                         default=2,
                         type=int,
                         help='Cell size controls the density of reconstructions'
@@ -227,12 +250,14 @@ def config():
     parser.add_argument('--texturing-data-term',
                         metavar='<string>',
                         default='gmi',
+                        choices=['gmi', 'area'],
                         help=('Data term: [area, gmi]. Default: '
                               '%(default)s'))
 
     parser.add_argument('--texturing-outlier-removal-type',
                         metavar='<string>',
                         default='gauss_clamping',
+                        choices=['none', 'gauss_clamping', 'gauss_damping'],
                         help=('Type of photometric outlier removal method: ' 
                               '[none, gauss_damping, gauss_clamping]. Default: '  
                               '%(default)s'))
@@ -296,6 +321,29 @@ def config():
                         help=('Orthophoto ground resolution in pixels/meter'
                               'Default: %(default)s'))
 
+    parser.add_argument('--orthophoto-target-srs',
+                        metavar="<EPSG:XXXX>",
+                        type=str,
+                        default=None,
+                        help='Target spatial reference for orthophoto creation. '
+                             'Not implemented yet.\n'
+                             'Default: %(default)s')
+
+    parser.add_argument('--orthophoto-no-tiled',
+                        action='store_true',
+                        default=False,
+                        help='Set this parameter if you want a stripped geoTIFF.\n'
+                             'Default: %(default)s')
+
+    parser.add_argument('--orthophoto-compression',
+                        metavar='<string>',
+                        type=str,
+                        choices=['JPEG', 'LZW', 'PACKBITS', 'DEFLATE', 'LZMA', 'NONE'],
+                        default='DEFLATE',
+                        help='Set the compression to use. Note that this could '
+                             'break gdal_translate if you don\'t know what you '
+                             'are doing. Options: %(choices)s.\nDefault: %(default)s')
+
     parser.add_argument('--zip-results',
                         action='store_true',
                         default=False,
@@ -313,4 +361,19 @@ def config():
                         help='Generates a benchmark file with runtime info\n'
                              'Default: %(default)s')
 
-    return parser.parse_args()
+    parser.add_argument('--version',
+                        action='version',
+                        version='OpenDroneMap {0}'.format(__version__),
+                        help='Displays version number and exits. ')
+
+    args = parser.parse_args()
+
+    # check that the project path setting has been set properly
+    if not args.project_path:
+        log.ODM_ERROR('You need to set the project path in the '
+                      'settings.yaml file before you can run ODM, '
+                      'or use `--project-path <path>`. Run `python '
+                      'run.py --help` for more information. ')
+        sys.exit(1)
+
+    return args

opendm/context.py

@@ -1,5 +1,6 @@
 import os
 import sys
+from opendm import io
 import multiprocessing
 
 # Define some needed locations
@@ -37,6 +38,8 @@ pdal_path = os.path.join(superbuild_path, 'build/pdal/bin')
 odm_modules_path = os.path.join(root_path, "build/bin")
 odm_modules_src_path = os.path.join(root_path, "modules")
 
+settings_path = os.path.join(root_path, 'settings.yaml')
+
 # Define supported image extensions
 supported_extensions = {'.jpg','.jpeg','.png'}
 

packages.Dockerfile

@@ -69,7 +69,8 @@ RUN pip install -U PyYAML \
                     exifread \
                     gpxpy \
                     xmltodict \
-                    catkin-pkg
+                    catkin-pkg \
+                    appsettings
 
 #Installing Ecto Dependencies
 RUN apt-get install -y -qq python-empy \

run.py

@@ -5,33 +5,24 @@ from opendm import config
 from opendm import system
 from opendm import io
 
-import sys
 import ecto
 import os
 
 from scripts.odm_app import ODMApp
 
-
-def usage():
-    log.ODM_ERROR('USAGE: %s --project-path [project_path]' % sys.argv[0])
-    log.ODM_ERROR('OpenDroneMap app finished - %s' % system.now())
-    sys.exit(0)
-
-
 if __name__ == '__main__':
 
-    log.ODM_INFO('Initializing OpenDroneMap app - %s' % system.now())
-
     args = config.config()
 
-    # Force to provide the images path
-    if args.project_path is None:
-        usage()
-    elif not io.dir_exists(args.project_path):
-        log.ODM_WARNING('Directory %s does not exist. Creating it now.' % args.project_path)
+    log.ODM_INFO('Initializing OpenDroneMap app - %s' % system.now())
+
+    # Add project dir if doesn't exist
+    args.project_path = io.join_paths(args.project_path, args.name)
+    if not io.dir_exists(args.project_path):
+        log.ODM_WARNING('Directory %s does not exist. Creating it now.' % args.name)
         system.mkdir_p(os.path.abspath(args.project_path))
 
-    #If user asks to rerun everything, delete all of the existing progress directories.
+    # If user asks to rerun everything, delete all of the existing progress directories.
     # TODO: Move this somewhere it's not hard-coded
     if args.rerun_all:
         os.system("rm -rf "

scripts/odm_app.py

@@ -73,6 +73,9 @@ class ODMApp(ecto.BlackBox):
                                                          use_exif=p.args.use_exif,
                                                          verbose=p.args.verbose),
                  'orthophoto': ODMOrthoPhotoCell(resolution=p.args.orthophoto_resolution,
+                                                 t_srs=p.args.orthophoto_target_srs,
+                                                 no_tiled=p.args.orthophoto_no_tiled,
+                                                 compress=p.args.orthophoto_compression,
                                                  verbose=p.args.verbose)
                  }
 

scripts/odm_orthophoto.py

@@ -10,6 +10,9 @@ from opendm import types
 class ODMOrthoPhotoCell(ecto.Cell):
     def declare_params(self, params):
         params.declare("resolution", 'Orthophoto ground resolution in pixels/meter', 20)
+        params.declare("t_srs", 'Target SRS', None)
+        params.declare("no_tiled", 'Do not tile tiff', False)
+        params.declare("compress", 'Compression type', 'DEFLATE')
         params.declare("verbose", 'print additional messages to console', False)
 
     def declare_io(self, params, inputs, outputs):
@@ -93,20 +96,26 @@ class ODMOrthoPhotoCell(ecto.Cell):
                         'uly': uly,
                         'lrx': lrx,
                         'lry': lry,
+                        'tiled': '' if self.params.no_tiled else '-co TILED=yes ',
+                        'compress': self.params.compress,
+                        'predictor': '-co PREDICTOR=2 ' if self.params.compress in
+                                                           ['LZW', 'DEFLATE'] else '',
                         'epsg': georef.epsg,
+                        't_srs': self.params.t_srs or "EPSG:{0}".format(georef.epsg),
                         'png': tree.odm_orthophoto_file,
                         'tiff': tree.odm_orthophoto_tif,
                         'log': tree.odm_orthophoto_tif_log
                     }
 
                     system.run('gdal_translate -a_ullr {ulx} {uly} {lrx} {lry} '
-                                '-co TILED=yes '
-                                '-co COMPRESS=DEFLATE '
-                                '-co PREDICTOR=2 '
-                                '-co BLOCKXSIZE=512 '
-                                '-co BLOCKYSIZE=512 '
-                                '-co NUM_THREADS=ALL_CPUS '
-                               '-a_srs \"EPSG:{epsg}\" {png} {tiff} > {log}'.format(**kwargs))
+                               '{tiled} '
+                               '-co COMPRESS={compress} '
+                               '{predictor} '
+                               '-co BLOCKXSIZE=512 '
+                               '-co BLOCKYSIZE=512 '
+                               '-co NUM_THREADS=ALL_CPUS '
+                               '-a_srs \"EPSG:{epsg}\" '
+                               '{png} {tiff} > {log}'.format(**kwargs))
                     geotiffcreated = True
                 if not geotiffcreated:
                     log.ODM_WARNING('No geo-referenced orthophoto created due '

settings.yaml

@@ -0,0 +1,49 @@
+---
+# A list of global configuration variables
+# Uncomment lines as needed to edit default settings.
+# Note this only works for settings with default values. Some commands like --rerun <module>
+# or --force-ccd n will have to be set in the command line (if you need to)
+
+# This line is really important to set up properly
+project_path: '' # Example: '/home/user/ODMProjects
+
+# The rest of the settings will default to the values set unless you uncomment and change them
+#resize_to: 2400
+#start_with: 'resize'
+#end_with: 'odm_orthophoto'
+#rerun_all: False
+#zip_results: False
+#verbose: False
+#time: False
+#opensfm_processes: 4 # by default this is set to $(nproc)
+#min_num_features: 4000
+#matcher_threshold: 2.0
+#matcher_ratio: 0.6
+#matcher_neighbors: 8
+#matcher_distance: 0
+#use_pmvs: False # The cmvs/pmvs settings only matter if 'Enabled' is set to True
+#cmvs_maximages: 500
+#pmvs_level: 1
+#pmvs_csize: 2
+#pmvs_threshold: 0.7
+#pmvs_wsize: 7
+#pmvs_min_images: 3
+#pmvs_num_cores: 4 # by default this is set to $(nproc)
+#mesh_size: 100000
+#mesh_octree_depth: 9
+#mesh_samples: 1.0
+#mesh_solver_divide: 9
+#texturing_data_term: 'gmi'
+#texturing_outlier_removal_type: 'gauss_clamping'
+#texturing_skip_visibility_test: False
+#texturing_skip_global_seam_leveling: False
+#texturing_skip_local_seam_leveling: False
+#texturing_skip_hole_filling: False
+#texturing_keep_unseen_faces: False
+#texturing_tone_mapping: 'none'
+#gcp: !!null  # YAML tag for None
+#use_exif: False # Set to True if you have a GCP file (it auto-detects) and want to use EXIF
+#orthophoto_resolution: 20.0 # Pixels/meter
+#orthophoto_target_srs: !!null # Currently does nothing
+#orthophoto_no_tiled: False
+#orthophoto_compression: DEFLATE # Options are [JPEG, LZW, PACKBITS, DEFLATE, LZMA, NONE] Don't change unless you know what you are doing