2015-11-15 07:56:36 +00:00
# ogn-python
2015-10-24 21:13:21 +00:00
2017-09-16 07:19:21 +00:00
[![Build Status ](https://travis-ci.org/glidernet/ogn-python.svg?branch=master )](https://travis-ci.org/glidernet/ogn-python)
[![Coverage Status ](https://img.shields.io/coveralls/glidernet/ogn-python.svg )](https://coveralls.io/r/glidernet/ogn-python)
2015-11-15 08:26:09 +00:00
2016-03-18 21:56:59 +00:00
A database backend for the [Open Glider Network ](http://wiki.glidernet.org/ ).
The ogn-python module saves all received beacons into a database with [SQLAlchemy ](http://www.sqlalchemy.org/ ).
It connects to the OGN aprs servers with [python-ogn-client ](https://github.com/glidernet/python-ogn-client ).
2020-12-15 11:40:12 +00:00
It requires [redis ](http://redis.io ), [PostgreSQL ](http://www.postgresql.org/ ), [PostGIS ](http://www.postgis.net/ ) and [TimescaleDB ](https://www.timescale.com ).
2016-01-12 18:31:08 +00:00
[Examples ](https://github.com/glidernet/ogn-python/wiki/Examples )
2016-03-18 21:54:22 +00:00
## Installation and Setup
2016-01-12 18:31:08 +00:00
1. Checkout the repository
2019-01-24 18:44:19 +00:00
```
git clone https://github.com/glidernet/ogn-python.git
2020-05-12 16:28:39 +00:00
cd ogn-python
2019-01-24 18:44:19 +00:00
```
2016-01-12 18:31:08 +00:00
2020-05-12 16:28:39 +00:00
2. Optional: Create and use a virtual environment
```
python3 -m venv my_environment
source my_environment/bin/activate
```
3. Install python requirements
2015-12-05 09:04:26 +00:00
```
pip install -r requirements.txt
```
2015-11-15 08:26:09 +00:00
2020-05-12 16:28:39 +00:00
4. Install [PostgreSQL ](http://www.postgresql.org/ ) with [PostGIS ](http://www.postgis.net/ ) and [TimescaleDB ](https://www.timescale.com ) Extension.
2019-01-24 18:44:19 +00:00
Create a database (use "ogn" as default, otherwise you have to modify the configuration, see below)
2016-06-16 08:58:03 +00:00
2021-06-18 15:38:16 +00:00
5. Install redis for asynchronous tasks (like database feeding, takeoff/landing-detection, ...)
2015-12-05 09:04:26 +00:00
```
apt-get install redis-server
```
2015-11-15 08:26:09 +00:00
2021-06-18 15:38:16 +00:00
6. Set the environment
Your environment variables must point to the configuration file and to the app path.
2019-04-29 20:42:39 +00:00
2021-06-18 15:38:16 +00:00
```
export OGN_CONFIG_MODULE="config.py"
export FLASK_APP=ogn_python.py
```
2019-04-29 20:42:39 +00:00
2020-05-12 16:28:39 +00:00
7. Create database
2015-12-05 09:04:26 +00:00
```
2021-06-18 15:38:16 +00:00
flask database init
2015-12-05 09:04:26 +00:00
```
2015-11-15 08:26:09 +00:00
2020-11-17 13:58:23 +00:00
8. Optional: Import world border dataset (needed if you want to know the country a receiver belongs to, etc.)
2019-01-24 18:44:19 +00:00
Get the [World Borders Dataset ](http://thematicmapping.org/downloads/world_borders.php ) and unpack it.
Then import it into your database (we use "ogn" as database name).
```
shp2pgsql -s 4326 TM_WORLD_BORDERS-0.3.shp world_borders_temp | psql -d ogn
psql -d ogn -c "INSERT INTO countries SELECT * FROM world_borders_temp;"
psql -d ogn -c "DROP TABLE world_borders_temp;"
```
2020-11-17 13:58:23 +00:00
9. Get world elevation data (needed for AGL calculation)
2019-03-17 14:09:01 +00:00
Sources: There are many sources for DEM data. It is important that the spatial reference system (SRID) is the same as the database which is 4326.
2019-03-18 07:37:19 +00:00
The [GMTED2010 Viewer ](https://topotools.cr.usgs.gov/gmted_viewer/viewer.htm ) provides data for the world with SRID 4326. Just download the data you need.
2019-01-24 18:44:19 +00:00
2020-11-17 13:58:23 +00:00
10. Import the GeoTIFF into the elevation table:
2019-01-24 18:44:19 +00:00
```
2020-10-27 19:46:14 +00:00
raster2pgsql *.tif -s 4326 -d -M -C -I -F -t 25x25 public.elevation | psql -d ogn
2019-01-24 18:44:19 +00:00
```
2019-03-30 16:50:29 +00:00
2020-11-17 13:58:23 +00:00
11. Import Airports (needed for takeoff and landing calculation). A cup file is provided under tests:
2019-03-30 16:50:29 +00:00
```
flask database import_airports tests/SeeYou.cup
```
2020-11-17 13:58:23 +00:00
12. Import DDB (needed for registration signs in the logbook).
2019-03-30 16:50:29 +00:00
```
flask database import_ddb
```
2019-09-13 17:11:20 +00:00
2020-11-17 13:58:23 +00:00
13. Optional: Use supervisord
2019-09-13 17:11:20 +00:00
You can use [Supervisor ](http://supervisord.org/ ) to control the complete system. In the directory deployment/supervisor
we have some configuration files to feed the database (ogn-feed), run the celery worker (celeryd), the celery beat
(celerybeatd), the celery monitor (flower), and the python wsgi server (gunicorn). All files assume that
we use a virtual environment in "/home/pi/ogn-python/venv". Please edit if necessary.
2016-09-23 18:13:23 +00:00
There is also a [Vagrant ](https://www.vagrantup.com/ ) environment for the development of ogn-python.
You can create and start this virtual machine with `vagrant up` and login with `vagrant ssh` .
The code of ogn-python will be available in the shared folder `/vagrant` .
2016-03-18 21:54:22 +00:00
## Usage
2016-01-12 18:31:08 +00:00
### Running the aprs client and task server
To schedule tasks like takeoff/landing-detection (`logbook.compute`),
[Celery ](http://www.celeryproject.org/ ) with [Redis ](http://www.redis.io/ ) is used.
The following scripts run in the foreground and should be deamonized
2015-11-15 08:26:09 +00:00
(eg. use [supervisord ](http://supervisord.org/ )).
2016-01-29 04:33:59 +00:00
- Start the aprs client
2016-01-12 18:31:08 +00:00
```
2021-06-18 15:38:16 +00:00
flask gateway run
2016-01-12 18:31:08 +00:00
```
2016-01-29 04:33:59 +00:00
- Start a task server (make sure redis is up and running)
2015-11-15 08:26:09 +00:00
2016-01-12 18:31:08 +00:00
```
2020-11-15 17:25:01 +00:00
celery -A celery_app worker -l info
2016-01-12 18:31:08 +00:00
```
2016-01-29 04:33:59 +00:00
- Start the task scheduler (make sure a task server is up and running)
```
2020-11-15 17:25:01 +00:00
celery -A celery_app beat -l info
2016-01-29 04:33:59 +00:00
```
2019-03-06 20:11:46 +00:00
### Flask - Command Line Interface
2015-11-15 08:26:09 +00:00
```
2019-03-06 20:11:46 +00:00
Usage: flask [OPTIONS] COMMAND [ARGS]...
A general utility script for Flask applications.
Provides commands from Flask, extensions, and the application. Loads the
application defined in the FLASK_APP environment variable, or from a
wsgi.py file. Setting the FLASK_ENV environment variable to 'development'
will enable debug mode.
$ export FLASK_APP=app.py
$ export FLASK_ENV=development
$ flask run
Options:
--version Show the flask version
--help Show this message and exit.
Commands:
database Database creation and handling.
db Perform database migrations.
export Export data in several file formats.
flights Create 2D flight paths from data.
gateway Connection to APRS servers.
2020-12-12 12:48:05 +00:00
logbook Handling of takeoff/landings and logbook data.
2019-03-06 20:11:46 +00:00
routes Show the routes for the app.
2020-12-12 12:48:05 +00:00
run Run a development server.
shell Run a shell in the app context.
2015-11-15 08:26:09 +00:00
```
2019-03-06 20:11:46 +00:00
Most commands are command groups, so if you execute this command you will get further (sub)commands.
2015-11-15 08:26:09 +00:00
2019-04-14 17:57:40 +00:00
### Available tasks
2015-11-15 08:26:09 +00:00
2020-12-15 11:40:12 +00:00
- `app.tasks.transfer_to_database` - Take sender and receiver messages from redis and put them into the db.
2020-11-15 17:25:01 +00:00
- `app.tasks.update_takeoff_landings` - Compute takeoffs and landings.
2020-12-15 11:40:12 +00:00
- `app.tasks.update_logbook` - Add/update logbook entries.
- `app.tasks.update_logbook_max_altitude` - Add max altitudes in logbook when flight is complete (takeoff and landing).
- `app.tasks.update_statistics` - Calculate several statistics (also the sender/receiver rankings).
- `app.tasks.import_ddb` - Import registered devices from the DDB.
2016-02-04 22:05:32 +00:00
2019-04-14 17:57:40 +00:00
If the task server is up and running, tasks could be started manually. Here we compute takeoffs and landings for the past 90 minutes:
2016-02-04 22:05:32 +00:00
```
python3
2020-12-15 11:40:12 +00:00
>>>from app.tasks import update_takeoff_landings
2019-04-29 20:42:39 +00:00
>>>update_takeoff_landings.delay(last_minutes=90)
2016-02-04 22:05:32 +00:00
```
2015-11-15 08:26:09 +00:00
2020-11-20 16:31:03 +00:00
or directly from command line:
```
2020-12-15 11:40:12 +00:00
celery -A celery_app call import_ddb
2020-11-20 16:31:03 +00:00
```
2020-11-17 13:58:23 +00:00
## Notes for Raspberry Pi
For matplotlib we need several apt packages installed:
2020-11-20 16:31:03 +00:00
```
2020-11-17 13:58:23 +00:00
apt install libatlas3-base libopenjp2-7 libtiff5
2020-11-20 16:31:03 +00:00
```
2020-11-17 13:58:23 +00:00
2016-01-12 18:31:08 +00:00
## License
Licensed under the [AGPLv3 ](LICENSE ).