OpenDroneMap-docs/source/tutorials.rst

450 wiersze
24 KiB
ReStructuredText
Czysty Zwykły widok Historia

.. Tutorials
2020-03-16 17:54:02 +00:00
#########
Tutorials
2020-03-16 17:47:32 +00:00
#########
Below you will find instructions for some common use cases.
2020-03-16 17:54:02 +00:00
*********************************
Creating High Quality Orthophotos
2020-03-16 17:47:32 +00:00
*********************************
.. figure:: images/orthophoto.png
:alt: image of OpenDroneMap orthophoto
:align: center
Without any parameter tweaks, ODM chooses a good compromise between quality, speed and memory usage. If you want to get higher quality results, you need to tweak some parameters:
* ``--orthophoto-resolution`` is the resolution of the orthophoto in cm/pixel. Decrease this value for a higher resolution result.
2020-03-19 19:21:10 +00:00
* ``--ignore-gsd`` is a flag that instructs ODM to skip certain memory and speed optimizations that directly affect the orthophoto. Using this flag will increase runtime and memory usage, but may produce sharper results.
* ``--texturing-nadir-weight`` should be increased to ``29-32`` in urban areas to reconstruct better edges of roofs. It should be decreased to ``0-6`` in grassy / flat areas.
* ``--texturing-data-term`` should be set to `area` in forest areas.
2020-08-07 22:16:39 +00:00
* ``--mesh-size`` should be increased to ``300000-600000`` and ``--mesh-octree-depth`` should be increased to ``10-11`` in urban areas to recreate better buildings / roofs.
2020-03-16 17:54:02 +00:00
**********************
Calibrating the Camera
2020-03-16 17:47:32 +00:00
**********************
Camera calibration is a special challenge with commodity cameras. Temperature changes, vibrations, focus, and other factors can affect the derived parameters with substantial effects on resulting data. Automatic or self calibration is possible and desirable with drone flights, but depending on the flight pattern, automatic calibration may not remove all distortion from the resulting products. James and Robson (2014) in their paper `Mitigating systematic error in topographic models derived from UAV and groundbased image networks <https://onlinelibrary.wiley.com/doi/full/10.1002/esp.3609>`_ address how to minimize the distortion from self-calibration.
.. figure:: images/msimbasi_bowling.png
:alt: image of lens distortion effect on bowling of data
:align: center
2019-09-24 13:43:35 +00:00
*Bowling effect on point cloud over 13,000+ image dataset collected by World Bank Tanzania over the flood prone Msimbasi Basin, Dar es Salaam, Tanzania.*
To mitigate this effect, there are a few options but the simplest are as follows: fly two patterns separated by 20°, and rather than having a nadir (straight down pointing) camera, use one that tilts forward by 5°.
.. figure:: images/flightplanning.gif
:alt: animation showing optimum
:align: center
:height: 480
:width: 640
2019-09-24 05:32:48 +00:00
As this approach to flying can be take longer than typical flights, a pilot or team can fly a small area using the above approach. OpenDroneMap will generate a calibration file called cameras.json that then can be imported to be used to calibrate another flight that is more efficiently flown.
Alternatively, the following experimental method can be applied: fly with much lower overlap, but two *crossgrid* flights (sometimes called crosshatch) separated by 20° with a 5° forward facing camera.
2019-09-28 22:31:16 +00:00
* Crossgrid overlap percentages can be lower than parallel flights. To get good 3D results, you will require 68% overlap and sidelap for an equivalent 83% overlap and sidelap.
* To get good 2D and 2.5D (digital elevation model) results, you will require 42% overlap and sidelap for an equivalent 70% overlap and sidelap.
.. figure:: images/rotation.gif
:alt: animation showing experimental optimum
:align: center
Vertically separated flight lines also improve accuracy, but less so than a camera that is forward facing by 5°.
.. figure:: images/forward_facing.png
:alt: figure showing effect of vertically separated flight lines and forward facing cameras on improving self calibration
:align: center
2019-09-30 01:47:18 +00:00
From James and Robson (2014), `CC BY 4.0 <https://creativecommons.org/licenses/by/4.0>`_
2020-03-16 17:54:02 +00:00
*********************************
Creating Digital Elevation Models
2020-03-16 17:47:32 +00:00
*********************************
By default ODM does not create DEMs. To create a digital terrain model, make sure to pass the ``--dtm`` flag. To create a digital surface model, be sure to pass the ``--dsm`` flag.
.. figure:: images/digitalsurfacemodel.png
:alt: image of OpenDroneMap derived digital surface model
:align: center
For DTM generation, a Simple Morphological Filter (smrf) is used to classify points in ground vs. non-ground and only the ground points are used. The ``smrf`` filter can be controlled via several parameters:
* ``--smrf-scalar`` scaling value. Increase this parameter for terrains with lots of height variation.
* ``--smrf-slope`` slope parameter, which is a measure of "slope tolerance". Increase this parameter for terrains with lots of height variation. Should be set to something higher than 0.1 and not higher than 1.2.
* ``--smrf-threshold`` elevation threshold. Set this parameter to the minimum height (in meters) that you expect non-ground objects to be.
* ``--smrf-window`` window radius parameter (in meters) that corresponds to the size of the largest feature (building, trees, etc.) to be removed. Should be set to a value higher than 10.
Changing these options can affect the result of DTMs significantly. The best source to read to understand how the parameters affect the output is to read the original paper `An improved simple morphological filter for the terrain classification of airborne LIDAR data <https://www.researchgate.net/publication/258333806_An_Improved_Simple_Morphological_Filter_for_the_Terrain_Classification_of_Airborne_LIDAR_Data>`_ (PDF freely available).
Overall the ``--smrf-threshold`` option has the biggest impact on results.
SMRF is good at avoiding Type I errors (small number of ground points mistakenly classified as non-ground) but only "acceptable" at avoiding Type II errors (large number non-ground points mistakenly classified as ground). This needs to be taken in consideration when generating DTMs that are meant to be used visually, since objects mistaken for ground look like artifacts in the final DTM.
.. figure:: images/smrf.png
:alt: image of lens distortion effect on bowling of data
:align: center
Two other important parameters affect DEM generation:
* ``--dem-resolution`` which sets the output resolution of the DEM raster (cm/pixel)
* ``--dem-gapfill-steps`` which determines the number of progressive DEM layers to use. For urban scenes increasing this value to `4-5` can help produce better interpolation results in the areas that are left empty by the SMRF filter.
Example of how to generate a DTM::
docker run -ti --rm -v /my/project:/datasets/code <my_odm_image> --project-path /datasets --dtm --dem-resolution 2 --smrf-threshold 0.4 --smrf-window 24
2020-03-16 17:54:02 +00:00
************
2019-09-24 14:22:56 +00:00
Using Docker
2020-03-16 17:47:32 +00:00
************
2019-09-24 14:22:56 +00:00
Since many users employ docker to deploy OpenDroneMap, it can be useful to understand some basic commands in order to interrogate the docker instances when things go wrong, or we are curious about what is happening. Docker is a containerized environment intended, among other things, to make it easier to deploy software independent of the local environment. In this way, it is similar to virtual machines.
A few simple commands can make our docker experience much better.
Listing Docker Machines
2020-03-16 17:47:32 +00:00
=======================
2019-09-24 14:22:56 +00:00
We can start by listing available docker machines on the current machine we are running as follows:
::
> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2518817537ce opendronemap/odm "bash" 36 hours ago Up 36 hours zen_wright
1cdc7fadf688 opendronemap/nodeodm "/usr/bin/nodejs /va…" 37 hours ago Up 37 hours 0.0.0.0:3000->3000/tcp flamboyant_dhawan
If we want to see machines that may not be running but still exist, we can add the `-a` flag:
::
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2518817537ce opendronemap/odm "bash" 36 hours ago Up 36 hours zen_wright
1cdc7fadf688 opendronemap/nodeodm "/usr/bin/nodejs /va…" 37 hours ago Up 37 hours 0.0.0.0:3000->3000/tcp flamboyant_dhawan
cd7b9585b8f6 opendronemap/odm "bash" 3 days ago Exited (1) 37 hours ago nostalgic_lederberg
e31010c00b9a opendronemap/odm "python /code/run.py…" 3 days ago Exited (2) 3 days ago suspicious_kepler
c44e0d0b8448 opendronemap/nodeodm "/usr/bin/nodejs /va…" 3 days ago Exited (0) 37 hours ago wonderful_burnell
Accessing logs on the instance
2020-03-16 17:47:32 +00:00
==============================
2019-09-24 14:22:56 +00:00
Using either the `CONTAINER ID` or the name, we can access any logs available on the machine as follows:
::
> docker logs 2518817537ce
This is likely to be unwieldy large, but we can use a pipe `|` character and other tools to extract just what we need from the logs. For example we can move through the log slowly using the `more` command:
::
> docker logs 2518817537ce | more
[INFO] DTM is turned on, automatically turning on point cloud classification
[INFO] Initializing OpenDroneMap app - Mon Sep 23 01:30:33 2019
[INFO] ==============
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
...
2019-09-24 14:46:34 +00:00
Pressing `Enter` or `Space`, arrow keys or `Page Up` or `Page Down` keys will now help us navigate through the logs. The lower case letter `Q` will let us escape back to the command line.
2019-09-24 14:22:56 +00:00
We can also extract just the end of the logs using the `tail` commmand as follows:
::
> docker logs 2518817537ce | tail -5
[INFO] Cropping /datasets/code/odm_orthophoto/odm_orthophoto.tif
[INFO] running gdalwarp -cutline /datasets/code/odm_georeferencing/odm_georeferenced_model.bounds.gpkg -crop_to_cutline -co NUM_THREADS=8 -co BIGTIFF=IF_SAFER -co BLOCKYSIZE=512 -co COMPRESS=DEFLATE -co BLOCKXSIZE=512 -co TILED=YES -co PREDICTOR=2 /datasets/code/odm_orthophoto/odm_orthophoto.original.tif /datasets/code/odm_orthophoto/odm_orthophoto.tif --config GDAL_CACHEMAX 48.95%
Using band 4 of source image as alpha.
Creating output file that is 111567P x 137473L.
Processing input file /datasets/code/odm_orthophoto/odm_orthophoto.original.tif.
The value `-5` tells the tail command to give us just the last 5 lines of the logs.
Command line access to instances
2020-03-16 17:47:32 +00:00
================================
2019-09-24 14:22:56 +00:00
Sometimes we need to go a little deeper in our exploration of the process for OpenDroneMap. For this, we can get direct command line access to the machines. For this, we can use `docker exec` to execute a `bash` command line shell in the machine of interest as follows:
::
2020-08-09 01:39:53 +00:00
2020-03-16 15:24:27 +00:00
> docker exec -ti 2518817537ce bash
root@2518817537ce:/code#
Now we are logged into our docker instance and can explore the machine.
Cleaning up after Docker
2020-03-16 17:47:32 +00:00
========================
2020-03-16 15:24:27 +00:00
Docker has a lamentable use of space and by default does not clean up excess data and machines when processes are complete. This can be advantageous if we need to access a process that has since terminated, but carries the burden of using increasing amounts of storage over time. Maciej Łebkowski has an `excellent overview of how to manage excess disk usage in docker <https://lebkowski.name/docker-volumes/>`_.
2019-09-24 14:22:56 +00:00
2020-03-16 17:54:02 +00:00
*************************************
2020-03-16 13:35:57 +00:00
Using ODM from low-bandwidth location
2020-03-16 17:47:32 +00:00
*************************************
2020-03-16 13:35:57 +00:00
What is this and who is it for?
2020-03-16 17:47:32 +00:00
===============================
2020-03-16 13:35:57 +00:00
2020-08-10 13:48:40 +00:00
Via `Ivan Gayton's repo <https://github.com/ivangayton/GDAL_scripts/>`_.
2020-03-16 17:59:34 +00:00
2020-03-16 13:35:57 +00:00
`OpenDroneMap <https://www.opendronemap.org/>`__ cant always be
effectively set up locally—it takes a fairly powerful machine to process
large datasets—so a cloud machine can sometimes be the answer for people
in the field. However, bandwidth is a problem in many low-income
settings. This constraint cant be solved completely, but the following
method does a reasonable job of reducing the bandwidth needed to process
drone imagery datasets on the cloud from African locations.
Here we present a tricky but workable process to create an OpenDroneMap
cloud machine (*not* CloudODM, mind you, just a cloud-based instance of
ODM that you run from the command line) and use it to remotely process
large photo sets. It requires familiarity with Unix command line use,
ssh, a Digital Ocean account (Amazon AWS would work as well, possibly
with slight differences in the setup), and a moderate level of general
computer literacy. If you arent fairly computer-savvy and willing to
fuss with a slightly tricky setup,
`CloudODM <https://www.opendronemap.org/cloudodm/>`__ is what you should
be looking at.
The whole process is mostly targeted at someone flying substantial
missions in an African or similar location looking to process data ASAP
while still in a field setting. Therefore it emphasizes a workflow
intended to reduce bandwidth/data transfer, rather than just the
simplest way of running ODM.
Steps
2020-03-16 17:47:32 +00:00
=====
2020-03-16 13:35:57 +00:00
Install
2020-03-16 17:47:32 +00:00
-------
2020-03-16 13:35:57 +00:00
- Create a Digital Ocean droplet with at least 4GB of RAM. Thatll cost
about $20/month. Less than 4GB of RAM and the install will probably
fail. When we actually run the ODM process well resize it to a much
larger—and more expensive—cloud machine, but between runs you can
downsize it between runs to the second-cheapest droplet which costs
only $10/month (the cheapest droplet, at $5/month, comes with such a
small drive that you cant downsize back to it).
2020-09-14 16:10:15 +00:00
- Should be an Ubuntu 18.04 instance to ensure dependency
2020-03-16 13:35:57 +00:00
compatibility
- Create a user with sudo privileges. `Digital Oceans insanely good
documentation <https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-16-04>`__
can help you figure this out. In our case we set up a user called
``odm``, so connecting to it is via the command
``ssh odm@xxx.xxx.xxx.xxx`` (where the xs stand for the IPv4
address of your server). If you want to follow this example
closely, *do* use the username ``odm``; then your install path
will be ``/home/odm/ODM/`` and will match all of the examples in
this document. -When you log into the server, it will offer you
the option to upgrade to Ubuntu 18.04, a more recent version.
Dont. ODM native install doesnt work smoothly on 18.04. Go ahead
and execute ``sudo apt update`` and ``sudo apt upgrade`` to ensure
your server isnt dangerously without updates, but stay with
2020-09-14 16:10:15 +00:00
Ubuntu 18.04.
2020-03-16 13:35:57 +00:00
- Download and install ODM on it from the `ODM
Github <https://github.com/OpenDroneMap/ODM>`__ (regular, not WebODM)
with the following commands:
::
git pull https://github.com/OpenDroneMap/ODM.git
cd ODM
bash configure.sh install
- If you do this from the default home folder of your user
(i.e. ``odm``) the path to the install will be ``/home/odm/ODM``
(abbreviated as ``~/ODM/``).
- There are some environmental variables that need to be set. Open the
~/.bashrc file on your machine and add the following 3 lines at the
end (From `the ODM github <https://github.com/OpenDroneMap/ODM>`__).
The file can be opened with ``nano ~/.bashrc`` (or whatever text
editor you use in lieu of nano). Be sure to replace ``/home/odm/``
with the correct path to the location where you extracted
OpenDroneMap if you didnt do everything exactly as in our example
(for example if you used a different username in your server setup):
::
export PYTHONPATH=$PYTHONPATH:/home/odm/ODM/SuperBuild/install/lib/python2.7/dist-packages
export PYTHONPATH=$PYTHONPATH:/home/odm/ODM/SuperBuild/src/opensfm
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/odm/ODM/SuperBuild/install/lib
- Note that the ODM github readme contains a slight error, the install
directory name will be ODM, not OpenDroneMap (youll see this if you
compare the above instructions to the ones on the ODM GitHub).
- In order to prevent a crash wherein the split-merge process fails to
locate its own executable, we add the following lines to
``~/.bashrc`` (adjust paths if youve set things up differently from
our example):
::
export PYTHONPATH=$PYTHONPATH:/home/odm/ODM/
export PATH=$PATH:/home/odm/ODM/
- Now youll need a second cloud hard drive (a “Volume” in Digital
Ocean jargon) big enough to manage your project. Rule of thumb seems
to be 10 times the size of your raw image set; weve got a 100GB
image set and set up a 1000GB volume (once the run is done you should
be able to get rid of most of this expensive drive capacity, but its
needed to complete the process). Set up the volume, attach it to your
droplet, and `configure its mount
point <https://www.digitalocean.com/docs/volumes/how-to/mount/>`__
(in this example were setting it to ``/mnt/odmdata/``).
Prep data and project
2020-03-16 17:47:32 +00:00
---------------------
2020-03-16 13:35:57 +00:00
- Now push your images onto the server. You can use `Secure Copy
(scp) <https://en.wikipedia.org/wiki/Secure_copy>`__ like so:
``scp -r /path/to/my/imagefolder odm@xxx.xxx.xxx.xxx:/mnt/odmdata/``.
- This pushes the entire folder full of images (thats what the
``-r`` option does, “recursive”) into the remote location (in our
example, into the volume we attached to the cloud machine at
``/mnt/odmdata/``.
- This will take some bandwidth. No way around the size of the
files.\ `1 <#footnote1>`__, \ `2 <#footnote2>`__\
Directory structure
2020-03-16 17:47:32 +00:00
^^^^^^^^^^^^^^^^^^^
2020-03-16 13:35:57 +00:00
ODM requires the directories on the machine to be set up just so. The
critical bits are the install folder (if you installed as above, its
``/home/odm/ODM/``) and the project folder
(i.e. ``/mnt/odmdata/myproject/``)
- ODMs settings.yaml file specifies a single parent directory
containing all projects. This is what goes in the project path line
of the settings.yaml file (slightly confusingly, this is actually the
*parent* directory of the individual project directories, which are
specified by the project name parameter when calling ODM). Edit
settings.yaml and set the project_path parameter to (as per our
example setup) ``/mnt/odmdata/``, which in this case points to the
Volume we created. Individual project directories are created within
that.
- Individual project directories, i.e. ``/mnt/odmdata/myproject/``
contain the gcp_list.txt file, the image_groups.txt file, and the
images folder for each project``\`
- The images folder, i.e. ``/mnt/odmdata/myproject/images/`` contains
all of the images. If you set it up like this, the images dont get
re-copied because theyre already in the directory that ODM wants
them in.
- Modify settings.yaml to specify the parent directory of the project
folder (in this case the Volume we created, ``/mnt/odmdata/``). Make
sure the images are in the correct spot,
i.e. ``/mnt/odmdata/myproject/images`` and the other ancillary files
(gcp_list.txt and image_groups.txt) are in the root folder
``/mnt/odmdata/myproject/``
- if you have the images in separate folders for individual AOI blocks
or flights (which you will if your flight management was organized),
you can create an image_groups.txt file with the incantations
``for i in *; do cd $i; for j in *; do echo "$j $i" >> ../$i.txt; done; cd ../; done;``
and ``cd ../``,
``for i in myproject/*.txt; do cat $i >> image_groups.txt; done;``.
That should create a file with the correct structure: a list of all
image files and a “group name” after each one (which in this case
will simply be the name of the folder it came from). Then move all of
the image files into a single directory called images in the project
root dir (so ``/mnt/odmdata/myproject/images/``). The
image_groups.txt file will allow ODM to keep track of which images
belong to the same batch, even though theyre all in a single
directory.
Resize droplet, pull pin, run away
2020-03-16 17:47:32 +00:00
----------------------------------
2020-03-16 13:35:57 +00:00
- Shut down and resize your machine to an appropriately monstrous
number of CPUs and amount of memory. I use the memory-optimized
machine with 24 dedicated vCPUs and 192GB of RAM (which costs about
$1.60/hr—which adds up fast, its over $1000/month). Restart, and get
to work quickly so as not to waste expensive big-droplet time.
- Launch the ODM process via ssh using nohup (so that if youre cut
off, processing will continue)
- Alternately you can use GNU screen to launch the process from a
screen session which wont stop if your connection is interrupted;
launch ``screen``, and use ``<ctrl> a <ctrl> d`` to detach,
``screen -r`` to re-attach. But using screen wont get you a log
file of all of the console output unless you do something specific
to capture that, while nohup gives you a file with all of the
console output, including error messages, for free.
- Note: as of 2020-03 the normal incantation
``python run.py -i /path/to/image/folder project_name`` seems
*not* to work; the ``-i`` or ``--image`` parameter causes a weird
error. So we drop the -i parameter, and rely on the project
directory line in the settings.yaml file to direct ODM to the
right place. Now using (including a split-merge):
::
nohup python run.py myproject --split 1 --split-overlap 0 --ignore-gsd --depthmap-resolution 1000 --orthophoto-resolution 5 --dem-resolution 15 --pc-las --dsm
- This points ODM at the folder (in this example)
``/mnt/odmdata/myproject/``. Provided the image_groups.txt and
gcp_list.txt are in this folder, the images are in
``/mnt/odmdata/myproject/images/``, and the project path in
settings.yaml is ``/mnt/odmdata/`` it will not waste time and space
copying images.
- Note that this assumes you have an image_groups.txt file. If not,
this ``-split-overlap 0`` will probably fuck things up, and the
``--split 1`` is literally a random number that will be ignored after
the image_groups.txt file is loaded (I think it normally controls how
many groups it splits a set of images into, but in our case were
assuming the images are already grouped sensibly). If you dont have
a large dataset (>1000 images), omit the ``--split`` and
``--split-overlap`` options.
- Follow the progress using tail (so that youll know when its done)
::
tail -f nohup.out
- You may want to keep an eye on htop (to get a sense of the resource
usage so that in future you can only spin up a machine as large as
necessary)
After it finishes (assuming you survive that long)
2020-03-16 17:47:32 +00:00
--------------------------------------------------
2020-03-16 13:35:57 +00:00
- As soon as processing is done, shut down the machine and resize it
back down to the inexpensive minimum capacity.
- Start the machine back up, and log in via ssh.
- If you want to save download bandwidth, you can compress the
orthophoto using GDAL. Dont add overviews, do that on your local
machine to avoid making the file bigger before downloading it.
::
gdal_translate -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -b 1 -b 2 -b 3 -mask 4 --config GDAL_TIFF_INTERNAL_MASK YES /path/to/original/filename.extension /path/to/output.tif
- Download using scp:
``scp odm@xxx.xxx.xxx.xxx:/mnt/odmdata/myproject/odm_orthophoto/odm_orthophoto.tif``
(or grab the compressed version you created in the last step)
- Once you get the file on your local computer, you can use QGIS to add
overviews (“pyramids”) or use the GDAL command
``gdaladdo -r average /path/to/image.tif 2 4 8 16 32 64 128 256 512 1024``.
- You can archive the odm_texturing, odm_georeferencing, and odm-dem
folders using tar to make them easier to download in one piece (and
maybe smaller).
::
tar -zcvf archivename /path/to/folder