db6d662dfa
Specify the user model using the AUTH_USER_MODEL setting as discussed in the Docs. https://docs.djangoproject.com/en/dev/topics/auth/customizing/#referencing-the-user-model |
||
---|---|---|
.github | ||
app | ||
build | ||
contrib | ||
db | ||
nginx | ||
nodeodm | ||
plugins | ||
screenshots | ||
service | ||
slate | ||
webodm | ||
worker | ||
.babelrc | ||
.env | ||
.gitignore | ||
.gitmodules | ||
.pyup.yml | ||
.travis.yml | ||
CONDUCT.md | ||
CONTRIBUTING.md | ||
Dockerfile | ||
ISSUE_TEMPLATE.md | ||
LICENSE.md | ||
README.md | ||
dev-worker.sh | ||
devenv.sh | ||
docker-compose.build.yml | ||
docker-compose.dev.yml | ||
docker-compose.nodemicmac.yml | ||
docker-compose.nodeodm.yml | ||
docker-compose.ssl-manual.yml | ||
docker-compose.ssl.yml | ||
docker-compose.yml | ||
jest.config.js | ||
manage.py | ||
package.json | ||
requirements.txt | ||
start.sh | ||
wait-for-it.sh | ||
wait-for-postgres.sh | ||
webodm.sh | ||
webpack-server.js | ||
webpack.config.js | ||
worker.sh |
README.md
A user-friendly, extendable application and API for drone image processing. Generate georeferenced maps, point clouds, elevation models and textured 3D models from aerial images. It supports multiple engines for processing, currently ODM and MicMac (experimental).
- Getting Started
- Recommended Machine Specs
- Customizing and Extending
- API Docs
- ODM, NodeODM, WebODM... what?
- Roadmap
- Getting Help
- Support the Project
- Become a Contributor
- Architecture Overview
- Run the docker version as a Linux Service
- Run it natively
- Run it on the cloud (Google Compute, Amazon AWS)
Getting Started
- Install the following applications (if they are not installed already):
-
Windows users have a choice between Docker Toolbox (Windows 10 Home or older) and Docker for Windows (Windows 10 Pro or newer). Docker for Windows users should set up their Docker environment before launching WebODM using the Docker utility in the system tray: 1) make sure Linux containers are enabled (Switch to Linux Containers...), 2) give Docker enough CPUs (default 2) and RAM (>4Gb, 16Gb better but leave some for Windows) by going to Settings -- Advanced, and 3) select where on your hard drive you want virtual hard drives to reside (Settings -- Advanced -- Images & Volumes).
-
From the Docker Quickstart Terminal or Powershell (Windows), or from the command line (Mac / Linux), type:
git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
cd WebODM
./webodm.sh start
- Open a Web Browser to
http://localhost:8000
(unless you are on Windows using Docker Toolbox, see below)
Docker Toolbox users need to find the IP of their docker machine by running this command from the Docker Quickstart Terminal:
docker-machine ip
192.168.1.100 (your output will be different)
The address to connect to would then be: http://192.168.1.100:8000
.
To stop WebODM press CTRL+C or run:
./webodm.sh stop
To update WebODM to the latest version use:
./webodm.sh update
We recommend that you read the Docker Documentation to familiarize with the application lifecycle, setup and teardown, or for more advanced uses. Look at the contents of the webodm.sh script to understand what commands are used to launch WebODM.
For Windows and macOS users an installer is also available.
You can also run WebODM from a Live USB/DVD. See LiveODM.
Manage Processing Nodes
WebODM can be linked to one or more processing nodes that speak the NodeODM API, such as NodeODM, NodeMICMAC or ClusterODM. The default configuration includes a "node-odm-1" processing node which runs on the same machine as WebODM, just to help you get started. As you become more familiar with WebODM, you might want to install processing nodes on separate machines.
Adding more processing nodes will allow you to run multiple jobs in parallel.
You can also setup a ClusterODM node to run a single task across multiple machines with distributed split-merge and process dozen of thousands of images more quickly, with less memory.
If you don't need the default "node-odm-1" node, simply pass the --no-default-node
flag when starting WebODM:
./webodm.sh restart --no-default-node
.
Then from the web interface simply manually remove the "node-odm-1" node.
Enable MicMac
WebODM can use MicMac as a processing engine via NodeMICMAC. To add MicMac, simply run:
./webodm.sh restart --with-micmac
This will create a "node-micmac-1" processing node on the same machine running WebODM. Please note that NodeMICMAC is in active development and is currently experimental. If you find issues, please report them on the NodeMICMAC repository.
Enable SSL
WebODM has the ability to automatically request and install a SSL certificate via Let’s Encrypt, or you can manually specify your own key/certificate pair.
- Setup your DNS record (webodm.myorg.com --> IP of server).
- Make sure port 80 and 443 are open.
- Run the following:
./webodm.sh restart --ssl --hostname webodm.myorg.com
That's it! The certificate will automatically renew when needed.
If you want to specify your own key/certificate pair, simply pass the --ssl-key
and --ssl-cert
option to ./webodm.sh
. See ./webodm.sh --help
for more information.
Where Are My Files Stored?
When using Docker, all processing results are stored in a docker volume and are not available on the host filesystem. If you want to store your files on the host filesystem instead of a docker volume, you need to pass a path via the --media-dir
option:
./webodm.sh restart --media-dir /home/user/webodm_data
Note that existing task results will not be available after the change. Refer to the Migrate Data Volumes section of the Docker documentation for information on migrating existing task results.
Common Troubleshooting
Sympthoms | Possible Solutions |
---|---|
While starting WebODM you get: from six.moves import _thread as thread ImportError: cannot import name _thread |
Try running: sudo pip install --ignore-installed six |
While starting WebODM you get: 'WaitNamedPipe','The system cannot find the file specified.' |
1. Make sure you have enabled VT-x virtualization in the BIOS. 2. Try to downgrade your version of Python to 2.7 |
While Accessing the WebODM interface you get: OperationalError at / could not translate host name “db” to address: Name or service not known or ProgrammingError at / relation “auth_user” does not exist |
Try restarting your computer, then type: ./webodm.sh restart |
Task output or console shows one of the following:
|
Make sure that your Docker environment has enough RAM allocated: MacOS Instructions, Windows Instructions |
After an update, you get: django.contrib.auth.models.DoesNotExist: Permission matching query does not exist. |
Try to remove your WebODM folder and start from a fresh git clone |
Task fails with Process exited with code null , no task console output - OR - console output shows Illegal Instruction - OR - console output shows Child returned 132 |
If the computer running NodeODM is using an old or 32bit CPU, you need to compile OpenDroneMap from sources and setup NodeODM natively. You cannot use docker. Docker images work with CPUs with 64-bit extensions, MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2 and POPCNT instruction set support or higher. |
On Windows, docker-compose fails with Failed to execute the script docker-compose |
Make sure you have enabled VT-x virtualization in the BIOS |
Cannot access WebODM using Microsoft Edge on Windows 10 | Try to tweak your internet properties according to these instructions |
Getting a No space left on device error, but hard drive has enough space left |
Docker on Windows by default will allocate only 20GB of space to the default docker-machine. You need to increase that amount. See this link and this link |
Cannot start WebODM via ./webodm.sh start , error messages are different at each retry |
You could be running out of memory. Make sure you have enough RAM available. 2GB should be the recommended minimum, unless you know what you are doing |
While running WebODM with Docker Toolbox (VirtualBox) you cannot access WebODM from another computer in the same network. | As Administrator, run cmd.exe and then type "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" controlvm "default" natpf1 "rule-name,tcp,,8000,,8000" |
Have you had other issues? Please report them so that we can include them in this document.
Backup and Restore
If you want to move WebODM to another system, you just need to transfer the docker volumes (unless you are storing your files on the file system).
On the old system:
mkdir -v backup
docker run --rm --volume webodm_dbdata:/temp --volume `pwd`/backup:/backup ubuntu tar cvf /backup/dbdata.tar /temp
docker run --rm --volume webodm_appmedia:/temp --volume `pwd`/backup:/backup ubuntu tar cvf /backup/appmedia.tar /temp
Your backup files will be stored in the newly created backup
directory. Transfer the backup
directory to the new system, then on the new system:
ls backup # --> appmedia.tar dbdata.tar
./webodm.sh start && ./webodm.sh down # Create volumes
docker run --rm --volume webodm_dbdata:/temp --volume `pwd`/backup:/backup ubuntu bash -c "rm -fr /temp/* && tar xvf /backup/dbdata.tar"
docker run --rm --volume webodm_appmedia:/temp --volume `pwd`/backup:/backup ubuntu bash -c "rm -fr /temp/* && tar xvf /backup/appmedia.tar"
./webodm.sh start
Reset Password
If you forgot the password you picked the first time you logged into WebODM, to reset it just type:
./webodm.sh start && ./webodm.sh resetadminpassword newpass
The password will be reset to newpass
. The command will also tell you what username you chose.
Manage Plugins
Plugins can be enabled and disabled from the user interface. Simply go to Administration -- Plugins.
Update
If you use docker, updating is as simple as running:
./webodm.sh update
If you are running WebODM natively, these commands should do it:
cd /webodm
sudo su odm # Only in case you are running WebODM with a different user
git pull origin master
source python3-venv/bin/activate # If you are running a virtualenv
npm install
pip install -r requirements.txt
webpack --mode production
python manage.py collectstatic --noinput
python manage.py migrate
Recommended Machine Specs
To run a standalone installation of WebODM (the user interface), including the processing component (NodeODM), we recommend at a minimum:
- 100 GB free disk space
- 16 GB RAM
Don't expect to process more than a few hundred images with these specifications. To process larger datasets, add more RAM linearly to the number of images you want to process. A CPU with more cores will speed up processing, but can increase memory usage. GPU acceleration is still a work in progress, so currently a good video card does not improve performance.
WebODM runs best on Linux, but works well on Windows and Mac too. If you are technically inclined, you can get WebODM to run natively on all three platforms and there's a native installer for Ubuntu 16.04 also available.
NodeODM and ODM cannot run natively on Mac and Windows and this is the reason we mostly recommend people to use docker.
WebODM by itself is just a user interface (see below) and does not require many resources. WebODM can be loaded on a machine with just 1 or 2 GB of RAM and work fine without NodeODM. You can then use a processing service such as the lightning network or run NodeODM on a separate, more powerful machine.
Customizing and Extending
Small customizations such as changing the application colors, name, logo, or addying custom CSS/HTML/Javascript can be performed directly from the Customize -- Brand/Theme panels within WebODM. No need to fork or change the code.
More advanced customizations can be achieved by writing plugins. This is the preferred way to add new functionality to WebODM since it requires less effort than maintaining a separate fork. The plugin system features server-side signals that can be used to be notified of various events, a ES6/React build system, a dynamic client-side API for adding elements to the UI, a built-in data store, an async task runner, a GRASS engine, hooks to add menu items and functions to rapidly inject CSS, Javascript and Django views.
The plugin system is still in beta. The best source of documentation currently is to look at existing code. If a particular hook / entrypoint for your plugin does not yet exist, request it. We are adding hooks and entrypoints as we go.
To create a plugin simply copy the plugins/test
plugin into a new directory (for example, plugins/myplugin
), then modify manifest.json
, plugin.py
and issue a ./webodm.sh restart
.
API Docs
See the API documentation page.
ODM, NodeODM, WebODM... what?
The OpenDroneMap project is composed of several components.
- ODM is a command line toolkit that processes aerial images. Users comfortable with the command line are probably OK using this component alone.
- NodeODM is a lightweight interface and API (Application Program Interface) built directly on top of ODM. Users not comfortable with the command line can use this interface to process aerial images and developers can use the API to build applications. Features such as user authentication, map displays, etc. are not provided.
- WebODM adds more features such as user authentication, map displays, 3D displays, a higher level API and the ability to orchestrate multiple processing nodes (run jobs in parallel). Processing nodes are simply servers running NodeODM.
In general, follow these guidelines to find out what you should use:
I am a... | Best choice |
---|---|
End user, I'm not really comfortable with the command line | WebODM |
End user, I like shell commands, I need to process images for myself. I use other software to display processing results | ODM |
End user, I can work with the command line, but I'd rather not. I use other software to display processing results | NodeODM |
End user, I need a drone mapping application for my organization that everyone can use. | WebODM |
Developer, I'm looking to build an app that displays map results and takes care of things like permissions | WebODM |
Developer, I'm looking to build an app that will stay behind a firewall and just needs raw results | NodeODM |
Roadmap
We follow a bottom-up approach to decide what new features are added to WebODM. User feedback guides us in the decision making process and we collect such feedback via improvement requests.
Don't see a feature that you want? Open a feature request or help us build it.
Sometimes we also prioritize work that has received financial backing. If your organization is in the position to financially support the development of a particular feature, get in touch and we'll make it happen.
Getting Help
We have several channels of communication for people to ask questions and to get involved with the community:
We also have a Gitter Chat, but the preferred way to communicate is via the OpenDroneMap Community Forum.
Support the Project
There are many ways to contribute back to the project:
- Help us test new and existing features and report bugs and feedback.
- Share your aerial datasets.
- Help answer questions on the community forum and chat.
- ⭐️ us on GitHub.
- Spread the word about WebODM and OpenDroneMap on social media.
- While we don't accept donations, you can purchase an installer or a book.
- Become a contributor 🤘
Become a Contributor
The easiest way to get started is to take a look at our list of outstanding issues and pick one. You can also fix/improve something entirely new based on your experience with WebODM. All ideas are considered and people of all skill levels are welcome to contribute.
You don't necessarily need to be a developer to become a contributor. We can use your help to write better documentation and improve the user interface texts and visuals.
If you know how to code, we primarily use Python (Django), Javascript (React), HTML and SCSS. See the Development Quickstart and Contributing documents for more information.
To make a contribution, you will need to open a pull request (here's how). To make changes to WebODM, make a clone of the repository and run ./webodm.sh start --dev
.
If you have questions visit us on the forum and we'll be happy to help you out with your first contribution.
Architecture Overview
WebODM is built with scalability and performance in mind. While the default setup places all databases and applications on the same machine, users can separate its components for increased performance (ex. place a Celery worker on a separate machine for running background tasks).
A few things to note:
- We use Celery workers to do background tasks such as resizing images and processing task results, but we use an ad-hoc scheduling mechanism to communicate with NodeODM (which processes the orthophotos, 3D models, etc.). The choice to use two separate systems for task scheduling is due to the flexibility that an ad-hoc mechanism gives us for certain operations (capture task output, persistent data and ability to restart tasks mid-way, communication via REST calls, etc.).
- If loaded on multiple machines, Celery workers should all share their
app/media
directory with the Django application (via network shares). You can manage workers via./worker.sh
Run the docker version as a Linux Service
If you wish to run the docker version with auto start/monitoring/stop, etc, as a systemd style Linux Service, a systemd unit file is included in the service folder of the repo.
This should work on any Linux OS capable of running WebODM, and using a SystemD based service daemon (such as Ubuntu 16.04 server for example).
This has only been tested on Ubuntu 16.04 server.
The following pre-requisites are required:
- Requires odm user
- Requires docker installed via system (ubuntu:
sudo apt-get install docker.io
) - Requires screen to be installed
- Requires odm user member of docker group
- Required WebODM directory checked out to /webodm
- Requires that /webodm is recursively owned by odm:odm
- Requires that a Python 3 environment is used at /webodm/python3-venv
If all pre-requisites have been met, and repository is checked out to /opt/WebODM folder, then you can use the following steps to enable and manage the service:
First, to install the service, and enable the services to run at startup from now on:
sudo systemctl enable /webodm/service/webodm-gunicorn.service
sudo systemctl enable /webodm/service/webodm-nginx.service
To manually start/stop the service:
sudo systemctl stop webodm-gunicorn
sudo systemctl start webodm-gunicorn
To manually check service status:
sudo systemctl status webodm-gunicorn
Run it natively
WebODM can run natively on Windows, MacOS and Linux.
Ubuntu 16.04 LTS users can refer to this community script to install WebODM natively on a new machine.
A commercial installer with technical support is also available.
To run WebODM, you will need to install:
- PostgreSQL (>= 9.5)
- PostGIS 2.3
- Python 3.5
- GDAL (>= 2.1)
- Node.js (>= 6.0)
- Nginx (Linux/MacOS) - OR - Apache + mod_wsgi (Windows)
- Redis (>= 2.6)
On Linux, make sure you have:
apt-get install binutils libproj-dev gdal-bin nginx
On Windows use the OSGeo4W installer to install GDAL. MacOS users can use:
brew install postgres postgis
Then these steps should be sufficient to get you up and running:
git clone --depth 1 https://github.com/OpenDroneMap/WebODM
Create a WebODM/webodm/local_settings.py
file containing your database settings:
DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': 'webodm_dev',
'USER': 'postgres',
'PASSWORD': 'postgres',
'HOST': 'localhost',
'PORT': '5432',
}
}
From psql or pgadmin, connect to PostgreSQL, create a new database (name it webodm_dev
), connect to it and set the postgis.enable_outdb_rasters and postgis.gdal_enabled_drivers settings:
ALTER SYSTEM SET postgis.enable_outdb_rasters TO True;
ALTER SYSTEM SET postgis.gdal_enabled_drivers TO 'GTiff';
Start the redis broker:
redis-server
Then:
pip install -r requirements.txt
sudo npm install -g webpack
sudo npm install -g webpack-cli
npm install
webpack --mode production
python manage.py collectstatic --noinput
chmod +x start.sh && ./start.sh --no-gunicorn
Finally, start at least one celery worker:
./worker.sh start
The start.sh
script will use Django's built-in server if you pass the --no-gunicorn
parameter. This is good for testing, but bad for production.
In production, if you have nginx installed, modify the configuration file in nginx/nginx.conf
to match your system's configuration and just run start.sh
without parameters.
Windows users should refer to this guide to install Apache + mod_wsgi and run gunicorn:
gunicorn webodm.wsgi --bind 0.0.0.0:8000 --preload
If you are getting a rt_raster_gdal_warp: Could not create GDAL transformation object for output dataset creation
, make sure that your PostGIS installation has PROJ support:
SELECT PostGIS_Full_Version();
You may also need to set the environment variable PROJSO to the .so or .dll projection library your PostGIS is using. This just needs to have the name of the file. So for example on Windows, you would in Control Panel -> System -> Environment Variables add a system variable called PROJSO and set it to libproj.dll (if you are using proj 4.6.1). You'll have to restart your PostgreSQL service/daemon after this change. http://postgis.net/docs/manual-2.0/RT_ST_Transform.html
If you are using Windows and are unable to go past the pip install -r requirements.txt
command because of an error regarding zlib and Pillow, manually edit the requirements.txt
file, remove the Pillow requirement and run:
easy_install pillow
pip install -r requirements.txt
On Windows make sure that all of your PATH environment variables are set properly. These commands:
python --version
pip --version
npm --version
gdalinfo --version
redis-server --version
Should all work without errors.
Run it on the cloud (Google Compute, Amazon AWS)
12 steps, to have WebODM running on a cloud instance.
These steps are for Google Cloud, but can also be used for Amazon AWS, and other cloud platforms with small modifications:
- Launch a Google Cloud instance of Ubuntu 18.0 LTS.
- Open the SSH terminal - Google offers SSH via the website.
- Run sudo apt-get update
- Run sudo apt-get upgrade
- Run sudo apt-get install docker-compose
- Run sudo apt-get install python-pip
- Run git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1
- cd WebODM (Linux is case sensitive)
- sudo ./webodm.sh start
- You now can access webodm via the public IP address for your google instance. Remember the default port of 8000.
- Check that your instance's firewall is allowing inbound TCP connections on port 8000! If you forget this step you will not be able to connect to WebODM.
- Open http://GooglepublicIPaddressforyourinstance:8000
To setup the firewall on Google Cloud, open the instance, on the middle of the instance settings page find NIC0. Open it, and then add the TCP Port 8000 for ingress, and egress on the Firewall.