mirror
/
ogn-python
kopia lustrzana https://github.com/glidernet/ogn-python
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność

Merge pull request #151 from Meisterschueler/hr 

Massive scheme modifications
pull/78/head
Meisterschueler 2020-11-22 09:03:09 +01:00 zatwierdzone przez GitHub
rodzic 695478654b b263b00f7c
commit 8fab57bcf1
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
133 zmienionych plików z 180538 dodań i 3933 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

33
.travis.yml
Wyświetl plik

@ -1,21 +1,39 @@
sudo: false
language: python
services:
- redis
- postgresql
addons:
postgresql: "11.2"
before_install:
- sudo service postgresql stop
- sudo apt-get update
- sudo apt-get remove -y postgresql\*
- sudo apt-get install postgresql-11-postgis-2.5 postgresql-client
- sudo sed -i 's/port = 5433/port = 5432/' /etc/postgresql/11/main/postgresql.conf
- sudo cp /etc/postgresql/{9.6,11}/main/pg_hba.conf
- sudo service postgresql start 11
install:
- pip install --upgrade pip
- pip install tox
before_script:
- sudo add-apt-repository ppa:timescale/timescaledb-ppa -y
- sudo apt-get update -q
- sudo apt-get install -y timescaledb-postgresql-11 timescaledb-tools
- sudo timescaledb-tune -yes -pg-version 11
- sudo service postgresql restart 11
- psql -U postgres -c 'CREATE DATABASE ogn_test;'
- psql -U postgres -c 'CREATE EXTENSION postgis;'
script:
- tox
addons:
postgresql: 9.6
apt:
packages:
- postgresql-9.6-postgis-2.4
matrix:
include:
@ -28,3 +46,8 @@ matrix:
after_success:
- tox -e codecov
after_failure:
- echo "Job failed..."
- sudo cat /var/log/syslog
- psql --version

63
README.md
Wyświetl plik

@ -34,7 +34,7 @@ It requires [PostgreSQL](http://www.postgresql.org/), [PostGIS](http://www.postg
4. Install [PostgreSQL](http://www.postgresql.org/) with [PostGIS](http://www.postgis.net/) and [TimescaleDB](https://www.timescale.com) Extension.
Create a database (use "ogn" as default, otherwise you have to modify the configuration, see below)
5. Optional: Install redis for asynchronous tasks (like takeoff/landing-detection)
5. Install redis for asynchronous tasks (like database feeding, takeoff/landing-detection, ...)
```
apt-get install redis-server
@ -53,13 +53,7 @@ It requires [PostgreSQL](http://www.postgresql.org/), [PostGIS](http://www.postg
./flask database init
```
8. Optional: Prepare tables for TimescaleDB
```
./flask database init_timescaledb
```
9. Optional: Import world border dataset (needed if you want to know the country a receiver belongs to, etc.)
8. Optional: Import world border dataset (needed if you want to know the country a receiver belongs to, etc.)
Get the [World Borders Dataset](http://thematicmapping.org/downloads/world_borders.php) and unpack it.
Then import it into your database (we use "ogn" as database name).
@ -69,40 +63,30 @@ It requires [PostgreSQL](http://www.postgresql.org/), [PostGIS](http://www.postg
psql -d ogn -c "DROP TABLE world_borders_temp;"
```
10. Get world elevation data (needed for AGL calculation)
9. Get world elevation data (needed for AGL calculation)
Sources: There are many sources for DEM data. It is important that the spatial reference system (SRID) is the same as the database which is 4326.
The [GMTED2010 Viewer](https://topotools.cr.usgs.gov/gmted_viewer/viewer.htm) provides data for the world with SRID 4326. Just download the data you need.
For Europe we can get the DEM as GeoTIFF files from the [European Environment Agency](https://land.copernicus.eu/imagery-in-situ/eu-dem/eu-dem-v1.1).
Because the SRID of these files is 3035 and we want 4326 we have to convert them (next step)
11. Optional: Convert the elevation data into correct SRID
We convert elevation from one SRID (here: 3035) to target SRID (4326):
10. Import the GeoTIFF into the elevation table:
```
gdalwarp -s_srs "EPSG:3035" -t_srs "EPSG:4326" source.tif target.tif
```
12. Import the GeoTIFF into the elevation table:
```
raster2pgsql -s 4326 -c -C -I -M -t 100x100 elevation_data.tif public.elevation | psql -d ogn
raster2pgsql *.tif -s 4326 -d -M -C -I -F -t 25x25 public.elevation | psql -d ogn
```
13. Import Airports (needed for takeoff and landing calculation). A cup file is provided under tests:
11. Import Airports (needed for takeoff and landing calculation). A cup file is provided under tests:
```
flask database import_airports tests/SeeYou.cup
```
14. Import DDB (needed for registration signs in the logbook).
12. Import DDB (needed for registration signs in the logbook).
```
flask database import_ddb
```
15. Optional: Use supervisord
13. Optional: Use supervisord
You can use [Supervisor](http://supervisord.org/) to control the complete system. In the directory deployment/supervisor
we have some configuration files to feed the database (ogn-feed), run the celery worker (celeryd), the celery beat
(celerybeatd), the celery monitor (flower), and the python wsgi server (gunicorn). All files assume that
@ -128,13 +112,13 @@ The following scripts run in the foreground and should be deamonized
- Start a task server (make sure redis is up and running)
```
celery -A app.collect worker -l info
celery -A celery_app worker -l info
```
- Start the task scheduler (make sure a task server is up and running)
```
celery -A app.collect beat -l info
celery -A celery_app beat -l info
```
### Flask - Command Line Interface
@ -173,14 +157,10 @@ Most commands are command groups, so if you execute this command you will get fu
### Available tasks
- `app.collect.celery.update_takeoff_landings` - Compute takeoffs and landings.
- `app.collect.celery.update_logbook_entries` - Add/update logbook entries.
- `app.collect.celery.update_logbook_max_altitude` - Add max altitudes in logbook when flight is complete (takeoff and landing).
- `app.collect.celery.import_ddb` - Import registered devices from the DDB.
- `app.collect.celery.update_receivers_country_code` - Update country code in receivers table if None.
- `app.collect.celery.purge_old_data` - Delete AircraftBeacons and ReceiverBeacons older than given 'age'.
- `app.collect.celery.update_stats` - Create stats and update receivers/devices with stats.
- `app.collect.celery.update_ognrange` - Create receiver coverage stats for Melissas ognrange.
- `app.tasks.update_takeoff_landings` - Compute takeoffs and landings.
- `app.tasks.celery.update_logbook_entries` - Add/update logbook entries.
- `app.tasks.celery.update_logbook_max_altitude` - Add max altitudes in logbook when flight is complete (takeoff and landing).
- `app.tasks.celery.import_ddb` - Import registered devices from the DDB.
If the task server is up and running, tasks could be started manually. Here we compute takeoffs and landings for the past 90 minutes:
@ -190,5 +170,18 @@ python3
>>>update_takeoff_landings.delay(last_minutes=90)
```
or directly from command line:
```
celery -A celery_app call takeoff_landings
```
## Notes for Raspberry Pi
For matplotlib we need several apt packages installed:
```
apt install libatlas3-base libopenjp2-7 libtiff5
```
## License
Licensed under the [AGPLv3](LICENSE).

41
app/__init__.py
Wyświetl plik

@ -1,36 +1,61 @@
import os
from flask import Flask
from flask_bootstrap import Bootstrap
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_caching import Cache
from celery import Celery
from flask_redis import FlaskRedis
from config import configs
bootstrap = Bootstrap()
db = SQLAlchemy()
migrate = Migrate()
cache = Cache()
celery = Celery(__name__, broker='redis://localhost:6379/0')
redis_client = FlaskRedis()
celery = Celery()
def create_app(config_name='development'):
def create_app(config_name='default'):
# Initialize Flask
app = Flask(__name__)
# Load the configuration
if config_name == 'testing':
app.config.from_object('app.config.test')
else:
app.config.from_object('app.config.default')
configuration = configs[config_name]
app.config.from_object(configuration)
app.config.from_envvar("OGN_CONFIG_MODULE", silent=True)
celery.config_from_object(app.config)
# Initialize other things
bootstrap.init_app(app)
db.init_app(app)
migrate.init_app(app, db)
cache.init_app(app)
redis_client.init_app(app)
init_celery(app)
register_blueprints(app)
return app
def register_blueprints(app):
from app.main import bp as bp_main
app.register_blueprint(bp_main)
return app
def init_celery(app=None):
app = app or create_app(os.getenv('FLASK_CONFIG') or 'default')
celery.conf.broker_url = app.config['BROKER_URL']
celery.conf.result_backend = app.config['CELERY_RESULT_BACKEND']
celery.conf.update(app.config)
class ContextTask(celery.Task):
"""Make celery tasks work with Flask app context"""
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery

103
app/backend/liveglidernet.py
Wyświetl plik

@ -1,103 +0,0 @@
from datetime import datetime, timezone
from app.model import AircraftBeacon, ReceiverBeacon
from app import db
def utc_to_local(utc_dt):
return utc_dt.replace(tzinfo=timezone.utc).astimezone(tz=None)
def encode(address):
return "xx" + address
def decode(code):
return code[2:9]
def rec(min_timestamp, min_online_timestamp):
last_seen_query = (
db.session.query(ReceiverBeacon).filter(ReceiverBeacon.timestamp > min_timestamp).order_by(ReceiverBeacon.receiver_id, ReceiverBeacon.timestamp).distinct(ReceiverBeacon.receiver_id)
)
lines = []
lines.append('<?xml version="1.0" encoding="UTF-8"?>')
lines.append("<markers>")
lines.append('<m e="0"/>')
for receiver_beacon in last_seen_query:
if receiver_beacon.location is None or receiver_beacon.name.startswith("FNB"):
continue
lines.append(
'<m a="{0}" b="{1:.7f}" c="{2:.7f}" d="{3:1d}"/>'.format(
receiver_beacon.name, receiver_beacon.location.latitude, receiver_beacon.location.longitude, receiver_beacon.timestamp < min_online_timestamp
)
)
lines.append("</markers>")
xml = "\n".join(lines)
return xml
def lxml(show_offline=False, lat_max=90, lat_min=-90, lon_max=180, lon_min=-180):
timestamp_range_filter = [db.between(AircraftBeacon.timestamp, datetime(2018, 7, 31, 11, 55, 0), datetime(2018, 7, 31, 12, 5, 0))]
last_seen_query = db.session.query(AircraftBeacon).filter(*timestamp_range_filter).order_by(AircraftBeacon.device_id, AircraftBeacon.timestamp).distinct(AircraftBeacon.device_id)
lines = list()
lines.append('<?xml version="1.0" encoding="UTF-8"?>')
lines.append("<markers>")
for aircraft_beacon in last_seen_query:
device = aircraft_beacon.device
code = encode(device.address)
if device.info:
if not device.info.tracked or not device.info.identified:
continue
if not device.info.competition:
competition = device.info.registration[-2:]
else:
competition = device.info.competition
if not device.info.registration:
registration = "???"
else:
registration = device.info.registration
address = device.address
else:
competition = ("_" + code[-2:]).lower()
registration = code
address = 0
elapsed_time = datetime.utcnow() - aircraft_beacon.timestamp
elapsed_seconds = int(elapsed_time.total_seconds())
lines.append(
' <m a="{0:.7f},{1:.7f},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11},{12},{13}"/>'.format(
aircraft_beacon.location.latitude,
aircraft_beacon.location.longitude,
competition,
registration,
int(aircraft_beacon.altitude),
utc_to_local(aircraft_beacon.timestamp).strftime("%H:%M:%S"),
elapsed_seconds,
int(aircraft_beacon.track),
int(aircraft_beacon.ground_speed),
int(aircraft_beacon.climb_rate * 10) / 10,
aircraft_beacon.aircraft_type,
aircraft_beacon.receiver_name,
address,
code,
)
)
lines.append("</markers>")
xml = "\n".join(lines)
return xml

14
app/backend/ognrange.py
Wyświetl plik

@ -1,8 +1,6 @@
import json
from datetime import datetime, timedelta
from sqlalchemy import func, case
from sqlalchemy.sql.expression import label
from app.model import Receiver, ReceiverCoverage
from app import db
@ -26,11 +24,11 @@ def stations2_filtered_pl(start, end):
query = (
db.session.query(
Receiver.name.label("s"),
label("lt", func.round(func.ST_Y(Receiver.location_wkt) * 10000) / 10000),
label("lg", func.round(func.ST_X(Receiver.location_wkt) * 10000) / 10000),
case([(Receiver.lastseen > last_10_minutes, "U")], else_="D").label("u"),
db.label("lt", db.func.round(db.func.ST_Y(Receiver.location_wkt) * 10000) / 10000),
db.label("lg", db.func.round(db.func.ST_X(Receiver.location_wkt) * 10000) / 10000),
db.case([(Receiver.lastseen > last_10_minutes, "U")], else_="D").label("u"),
Receiver.lastseen.label("ut"),
label("v", Receiver.version + "." + Receiver.platform),
db.label("v", Receiver.version + "." + Receiver.platform),
)
.order_by(Receiver.lastseen)
.filter(db.or_(db.and_(start < Receiver.firstseen, end > Receiver.firstseen), db.and_(start < Receiver.lastseen, end > Receiver.lastseen)))
@ -44,10 +42,10 @@ def stations2_filtered_pl(start, end):
def max_tile_mgrs_pl(station, start, end, squares):
query = (
db.session.query(func.right(ReceiverCoverage.location_mgrs_short, 4), func.count(ReceiverCoverage.location_mgrs_short))
db.session.query(db.func.right(ReceiverCoverage.location_mgrs_short, 4), db.func.count(ReceiverCoverage.location_mgrs_short))
.filter(db.and_(Receiver.id == ReceiverCoverage.receiver_id, Receiver.name == station))
.filter(ReceiverCoverage.location_mgrs_short.like(squares + "%"))
.group_by(func.right(ReceiverCoverage.location_mgrs_short, 4))
.group_by(db.func.right(ReceiverCoverage.location_mgrs_short, 4))
)
res = {"t": squares, "p": ["{}/{}".format(r[0], r[1]) for r in query.all()]}

106
app/collect/celery.py
Wyświetl plik

@ -1,106 +0,0 @@
import datetime
from celery.utils.log import get_task_logger
from app.collect.takeoff_landings import update_entries as takeoff_update_entries
from app.collect.logbook import update_entries as logbook_update_entries
from app.collect.logbook import update_max_altitudes as logbook_update_max_altitudes
from app.collect.database import import_ddb as device_infos_import_ddb
from app.collect.database import update_country_code as receivers_update_country_code
from app.collect.stats import create_device_stats, update_device_stats_jumps, create_receiver_stats, create_relation_stats, update_qualities, update_receivers, update_devices
from app.collect.ognrange import update_entries as receiver_coverage_update_entries
from app import db
from app import celery
logger = get_task_logger(__name__)
@celery.task(name="update_takeoff_landings")
def update_takeoff_landings(last_minutes):
"""Compute takeoffs and landings."""
end = datetime.datetime.utcnow()
start = end - datetime.timedelta(minutes=last_minutes)
result = takeoff_update_entries(session=db.session, start=start, end=end, logger=logger)
return result
@celery.task(name="update_logbook_entries")
def update_logbook_entries(day_offset):
"""Add/update logbook entries."""
date = datetime.datetime.today() + datetime.timedelta(days=day_offset)
result = logbook_update_entries(session=db.session, date=date, logger=logger)
return result
@celery.task(name="update_logbook_max_altitude")
def update_logbook_max_altitude(day_offset):
"""Add max altitudes in logbook when flight is complete (takeoff and landing)."""
date = datetime.datetime.today() + datetime.timedelta(days=day_offset)
result = logbook_update_max_altitudes(session=db.session, date=date, logger=logger)
return result
@celery.task(name="import_ddb")
def import_ddb():
"""Import registered devices from the DDB."""
result = device_infos_import_ddb(session=db.session, logger=logger)
return result
@celery.task(name="update_receivers_country_code")
def update_receivers_country_code():
"""Update country code in receivers table if None."""
result = receivers_update_country_code(session=db.session, logger=logger)
return result
@celery.task(name="purge_old_data")
def purge_old_data(max_hours):
"""Delete AircraftBeacons and ReceiverBeacons older than given 'age'."""
from app.model import AircraftBeacon, ReceiverBeacon
min_timestamp = datetime.datetime.utcnow() - datetime.timedelta(hours=max_hours)
aircraft_beacons_deleted = db.session.query(AircraftBeacon).filter(AircraftBeacon.timestamp < min_timestamp).delete()
receiver_beacons_deleted = db.session.query(ReceiverBeacon).filter(ReceiverBeacon.timestamp < min_timestamp).delete()
db.session.commit()
result = "{} AircraftBeacons deleted, {} ReceiverBeacons deleted".format(aircraft_beacons_deleted, receiver_beacons_deleted)
return result
@celery.task(name="update_stats")
def update_stats(day_offset):
"""Create stats and update receivers/devices with stats."""
date = datetime.datetime.today() + datetime.timedelta(days=day_offset)
create_device_stats(session=db.session, date=date)
update_device_stats_jumps(session=db.session, date=date)
create_receiver_stats(session=db.session, date=date)
create_relation_stats(session=db.session, date=date)
update_qualities(session=db.session, date=date)
update_receivers(session=db.session)
update_devices(session=db.session)
@celery.task(name="update_ognrange")
def update_ognrange(day_offset):
"""Create receiver coverage stats for Melissas ognrange."""
date = datetime.datetime.today() + datetime.timedelta(days=day_offset)
receiver_coverage_update_entries(session=db.session, date=date)

47
app/collect/database.py
Wyświetl plik

@ -1,12 +1,12 @@
from sqlalchemy.sql import null, and_, func, case
from sqlalchemy.dialects.postgresql import insert
from flask import current_app
from app.model import Country, DeviceInfo, DeviceInfoOrigin, Receiver
from app import db
from app.model import SenderInfo, SenderInfoOrigin, Receiver
from app.utils import get_ddb, get_flarmnet
def upsert(session, model, rows, update_cols):
def upsert(model, rows, update_cols):
"""Insert rows in model. On conflicting update columns if new value IS NOT NULL."""
table = model.__table__
@ -14,59 +14,40 @@ def upsert(session, model, rows, update_cols):
stmt = insert(table).values(rows)
on_conflict_stmt = stmt.on_conflict_do_update(
index_elements=table.primary_key.columns, set_={k: case([(getattr(stmt.excluded, k) != null(), getattr(stmt.excluded, k))], else_=getattr(model, k)) for k in update_cols}
index_elements=table.primary_key.columns, set_={k: db.case([(getattr(stmt.excluded, k) != db.null(), getattr(stmt.excluded, k))], else_=getattr(model, k)) for k in update_cols}
)
# print(compile_query(on_conflict_stmt))
session.execute(on_conflict_stmt)
return on_conflict_stmt
def update_device_infos(session, address_origin, path=None):
if address_origin == DeviceInfoOrigin.FLARMNET:
def update_device_infos(address_origin, path=None):
if address_origin == SenderInfoOrigin.FLARMNET:
device_infos = get_flarmnet(fln_file=path)
else:
device_infos = get_ddb(csv_file=path)
session.query(DeviceInfo).filter(DeviceInfo.address_origin == address_origin).delete(synchronize_session="fetch")
session.commit()
db.session.query(SenderInfo).filter(SenderInfo.address_origin == address_origin).delete(synchronize_session="fetch")
db.session.commit()
for device_info in device_infos:
device_info.address_origin = address_origin
session.bulk_save_objects(device_infos)
session.commit()
db.session.bulk_save_objects(device_infos)
db.session.commit()
return len(device_infos)
def import_ddb(session, logger=None):
def import_ddb(logger=None):
"""Import registered devices from the DDB."""
if logger is None:
logger = current_app.logger
logger.info("Import registered devices fom the DDB...")
counter = update_device_infos(session, DeviceInfoOrigin.OGN_DDB)
counter = update_device_infos(SenderInfoOrigin.OGN_DDB)
finish_message = "DeviceInfo: {} inserted.".format(counter)
logger.info(finish_message)
return finish_message
def update_country_code(session, logger=None):
"""Update country code in receivers table if None."""
if logger is None:
logger = current_app.logger
update_receivers = (
session.query(Receiver)
.filter(and_(Receiver.country_id == null(), Receiver.location_wkt != null(), func.st_within(Receiver.location_wkt, Country.geom)))
.update({Receiver.country_id: Country.gid}, synchronize_session="fetch")
)
session.commit()
finish_message = "Receivers (country): {} updated".format(update_receivers)
finish_message = "SenderInfo: {} inserted.".format(counter)
logger.info(finish_message)
return finish_message

116
app/collect/flights.py 100644
Wyświetl plik

@ -0,0 +1,116 @@
from datetime import date
from app import db
NOTHING = ""
CONTEST_RELEVANT = "AND agl < 1000"
LOW_PASS = "AND agl < 50 and ground_speed > 250"
def compute_flights(date, flight_type=0):
if flight_type == 0:
filter = NOTHING
elif flight_type == 1:
filter = CONTEST_RELEVANT
elif flight_type == 2:
filter = LOW_PASS
date_str = date.strftime("%Y-%m-%d")
query = f"""
INSERT INTO flights(date, sender_id, flight_type, multilinestring, simple_multilinestring)
SELECT '{date_str}' AS date,
s.id AS sender_id,
{flight_type} as flight_type,
st_collect(sq5.linestring order BY sq5.part) multilinestring,
st_collect(st_simplify(sq5.linestring, 0.0001) ORDER BY sq5.part) simple_multilinestring
FROM (
SELECT sq4.name,
sq4.part,
st_makeline(sq4.location ORDER BY sq4.timestamp) AS linestring
FROM (
SELECT sq3.timestamp,
sq3.location,
sq3.name,
SUM(sq3.ping) OVER (partition BY sq3.name ORDER BY sq3.timestamp) AS part
FROM (
SELECT sq2.t1 AS timestamp,
sq2.l1 AS location,
sq2.s1 AS name,
CASE
WHEN sq2.s1 = sq2.s2 AND sq2.t1 - sq2.t2 < interval'100s' AND ST_DistanceSphere(sq2.l1, sq2.l2) < 1000 THEN 0
ELSE 1
END AS ping
FROM (
SELECT sq.timestamp t1,
lag(sq.timestamp) OVER (partition BY sq.name ORDER BY sq.timestamp) t2,
sq.location l1,
lag(sq.location) OVER (partition BY sq.name ORDER BY sq.timestamp) l2,
sq.name s1,
lag(sq.name) OVER (partition BY sq.name ORDER BY sq.timestamp) s2
FROM (
SELECT DISTINCT ON (name, timestamp) name, timestamp, location
FROM sender_positions
WHERE reference_timestamp BETWEEN '{date_str} 00:00:00' AND '{date_str} 23:59:59' {filter}
ORDER BY name, timestamp, error_count
) AS sq
) AS sq2
) AS sq3
) AS sq4
GROUP BY sq4.name, sq4.part
) AS sq5
INNER JOIN senders AS s ON sq5.name = s.name
GROUP BY s.id
ON CONFLICT DO NOTHING;
"""
db.session.execute(query)
db.session.commit()
def compute_gaps(date):
date_str = date.strftime("%Y-%m-%d")
query = f"""
INSERT INTO flights(date, flight_type, sender_id, multilinestring)
SELECT '{date_str}' AS date,
3 AS flight_type,
s.id AS sender_id,
ST_Collect(sq3.path)
FROM (
SELECT sq2.s1 AS name,
ST_MakeLine(sq2.l1, sq2.l2) AS path
FROM
(
SELECT sq.timestamp t1,
LAG(sq.timestamp) OVER (PARTITION BY sq.timestamp::DATE, sq.name ORDER BY sq.timestamp) t2,
sq.location l1,
LAG(sq.location) OVER (PARTITION BY sq.timestamp::DATE, sq.name ORDER BY sq.timestamp) l2,
sq.name s1,
LAG(sq.name) OVER (PARTITION BY sq.timestamp::DATE, sq.name ORDER BY sq.timestamp) s2
FROM
(
SELECT DISTINCT ON (name, timestamp) name, timestamp, location, agl
FROM sender_positions
WHERE reference_timestamp BETWEEN '{date_str} 00:00:00' AND '{date_str} 23:59:59' AND agl > 300
ORDER BY name, timestamp, error_count
) AS sq
) AS sq2
WHERE EXTRACT(epoch FROM sq2.t1 - sq2.t2) > 300
AND ST_DistanceSphere(sq2.l1, sq2.l2) / EXTRACT(epoch FROM sq2.t1 - sq2.t2) BETWEEN 15 AND 50
) AS sq3
INNER JOIN senders AS s on sq3.name = s.name
GROUP BY s.id
ON CONFLICT DO NOTHING;
"""
db.session.execute(query)
db.session.commit()
if __name__ == '__main__':
from app import create_app
app = create_app()
with app.app_context():
result = compute_flights(date=date(2020, 10, 28))
print(result)

24
app/collect/gateway.py 100644
Wyświetl plik

@ -0,0 +1,24 @@
from datetime import datetime
from flask import current_app
from app import redis_client
from app.gateway.message_handling import sender_position_csv_strings_to_db, receiver_position_csv_strings_to_db, receiver_status_csv_strings_to_db
def transfer_from_redis_to_database():
def unmapping(string):
return string[0].decode('utf-8')
receiver_status_data = list(map(unmapping, redis_client.zpopmin('receiver_status', 100000)))
receiver_position_data = list(map(unmapping, redis_client.zpopmin('receiver_position', 100000)))
sender_status_data = list(map(unmapping, redis_client.zpopmin('sender_status', 100000)))
sender_position_data = list(map(unmapping, redis_client.zpopmin('sender_position', 100000)))
receiver_status_csv_strings_to_db(lines=receiver_status_data)
receiver_position_csv_strings_to_db(lines=receiver_position_data)
sender_position_csv_strings_to_db(lines=sender_position_data)
current_app.logger.debug(f"transfer_from_redis_to_database: rx_stat: {len(receiver_status_data):6d}\trx_pos: {len(receiver_position_data):6d}\ttx_stat: {len(sender_status_data):6d}\ttx_pos: {len(sender_position_data):6d}")
finish_message = f"Database: {len(receiver_status_data)+len(receiver_position_data)+len(sender_status_data)+len(sender_position_data)} inserted"
return finish_message

469
app/collect/logbook.py
Wyświetl plik

@ -1,199 +1,382 @@
from sqlalchemy import and_, or_, insert, update, exists, between
from sqlalchemy.sql import func, null
from sqlalchemy.sql.expression import true, false
from sqlalchemy.dialects.postgresql import insert # special insert for upsert ("ON CONFLICT ...")
from flask import current_app
from app.model import TakeoffLanding, Logbook, AircraftBeacon
from app.model import Airport, Country, SenderPosition, Sender, TakeoffLanding, Logbook
from app.utils import date_to_timestamps
from datetime import datetime, timedelta
def update_entries(session, date, logger=None):
"""Add/update logbook entries."""
from app import db
if logger is None:
logger = current_app.logger
logger.info("Compute logbook.")
# takeoff / landing detection is based on 3 consecutive points
MIN_TAKEOFF_SPEED = 55 # takeoff detection: 1st point below, 2nd and 3rd above this limit
MAX_LANDING_SPEED = 40 # landing detection: 1st point above, 2nd and 3rd below this limit
MIN_TAKEOFF_CLIMB_RATE = -5 # takeoff detection: glider should not sink too much
MAX_LANDING_SINK_RATE = 5 # landing detection: glider should not climb too much
MAX_EVENT_DURATION = 100 # the points must not exceed this duration
MAX_EVENT_RADIUS = 5000 # the points must not exceed this radius around the 2nd point
MAX_EVENT_AGL = 200 # takeoff / landing must not exceed this altitude AGL
# limit time range to given date and set window partition and window order
(start, end) = date_to_timestamps(date)
pa = TakeoffLanding.device_id
wo = and_(TakeoffLanding.device_id, TakeoffLanding.airport_id, TakeoffLanding.timestamp)
# make a query with current, previous and next "takeoff_landing" event, so we can find complete flights
def update_takeoff_landings(start, end):
"""Compute takeoffs and landings."""
current_app.logger.info("Compute takeoffs and landings.")
# considered time interval should not exceed a complete day
if end - start > timedelta(days=1):
abort_message = "TakeoffLanding: timeinterval start='{}' and end='{}' is too big.".format(start, end)
current_app.logger.warn(abort_message)
return abort_message
# check if we have any airport
airports_query = db.session.query(Airport).limit(1)
if not airports_query.all():
abort_message = "TakeoffLanding: Cannot calculate takeoff and landings without any airport! Please import airports first."
current_app.logger.warn(abort_message)
return abort_message
# get beacons for selected time range (+ buffer for duration), one per name and timestamp
sq = (
session.query(
TakeoffLanding.device_id,
func.lag(TakeoffLanding.device_id).over(partition_by=pa, order_by=wo).label("device_id_prev"),
func.lead(TakeoffLanding.device_id).over(partition_by=pa, order_by=wo).label("device_id_next"),
TakeoffLanding.timestamp,
func.lag(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_prev"),
func.lead(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_next"),
TakeoffLanding.track,
func.lag(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_prev"),
func.lead(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_next"),
TakeoffLanding.is_takeoff,
func.lag(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_prev"),
func.lead(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_next"),
TakeoffLanding.airport_id,
func.lag(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_prev"),
func.lead(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_next"),
)
.filter(between(TakeoffLanding.timestamp, start, end))
db.session.query(SenderPosition.name, SenderPosition.timestamp, SenderPosition.location, SenderPosition.track, db.func.coalesce(SenderPosition.ground_speed, 0.0).label("ground_speed"), SenderPosition.altitude, db.func.coalesce(SenderPosition.climb_rate, 0.0).label("climb_rate"))
.distinct(SenderPosition.name, SenderPosition.timestamp)
.order_by(SenderPosition.name, SenderPosition.timestamp, SenderPosition.error_count)
.filter(SenderPosition.agl <= MAX_EVENT_AGL)
.filter(db.between(SenderPosition.reference_timestamp, start - timedelta(seconds=MAX_EVENT_DURATION), end + timedelta(seconds=MAX_EVENT_DURATION)))
.subquery()
)
# find complete flights
complete_flight_query = session.query(
sq.c.timestamp.label("reftime"),
sq.c.device_id.label("device_id"),
sq.c.timestamp.label("takeoff_timestamp"),
sq.c.track.label("takeoff_track"),
sq.c.airport_id.label("takeoff_airport_id"),
sq.c.timestamp_next.label("landing_timestamp"),
sq.c.track_next.label("landing_track"),
sq.c.airport_id_next.label("landing_airport_id"),
).filter(and_(sq.c.is_takeoff == true(), sq.c.is_takeoff_next == false()))
# make a query with current, previous and next position
sq2 = db.session.query(
sq.c.name,
db.func.lag(sq.c.name).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("name_prev"),
db.func.lead(sq.c.name).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("name_next"),
sq.c.timestamp,
db.func.lag(sq.c.timestamp).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("timestamp_prev"),
db.func.lead(sq.c.timestamp).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("timestamp_next"),
sq.c.location,
db.func.lag(sq.c.location).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("location_wkt_prev"),
db.func.lead(sq.c.location).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("location_wkt_next"),
sq.c.track,
db.func.lag(sq.c.track).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("track_prev"),
db.func.lead(sq.c.track).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("track_next"),
sq.c.ground_speed,
db.func.lag(sq.c.ground_speed).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("ground_speed_prev"),
db.func.lead(sq.c.ground_speed).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("ground_speed_next"),
sq.c.altitude,
db.func.lag(sq.c.altitude).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("altitude_prev"),
db.func.lead(sq.c.altitude).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("altitude_next"),
sq.c.climb_rate,
db.func.lag(sq.c.climb_rate).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("climb_rate_prev"),
db.func.lead(sq.c.climb_rate).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("climb_rate_next"),
).subquery()
# find landings without start
# consider only positions between start and end and with predecessor and successor and limit distance and duration between points
sq3 = (
db.session.query(sq2)
.filter(db.and_(sq2.c.name_prev != db.null(), sq2.c.name_next != db.null()))
.filter(db.and_(db.func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_prev) < MAX_EVENT_RADIUS, db.func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_next) < MAX_EVENT_RADIUS))
.filter(sq2.c.timestamp_next - sq2.c.timestamp_prev < timedelta(seconds=MAX_EVENT_DURATION))
.filter(db.between(sq2.c.timestamp, start, end))
.subquery()
)
# find possible takeoffs and landings
sq4 = (
db.session.query(
sq3.c.timestamp,
db.case(
[
(sq3.c.ground_speed > MIN_TAKEOFF_SPEED, sq3.c.location_wkt_prev), # on takeoff we take the location from the previous fix because it is nearer to the airport
(sq3.c.ground_speed <= MIN_TAKEOFF_SPEED, sq3.c.location),
]
).label("location"),
db.case([(sq3.c.ground_speed > MAX_LANDING_SPEED, sq3.c.track), (sq3.c.ground_speed <= MAX_LANDING_SPEED, sq3.c.track_prev)]).label(
"track"
), # on landing we take the track from the previous fix because gliders tend to leave the runway quickly
sq3.c.ground_speed,
sq3.c.altitude,
db.case([(sq3.c.ground_speed > MIN_TAKEOFF_SPEED, True), (sq3.c.ground_speed < MAX_LANDING_SPEED, False)]).label("is_takeoff"),
sq3.c.name,
)
.filter(
db.or_(
db.and_(sq3.c.ground_speed_prev < MIN_TAKEOFF_SPEED, sq3.c.ground_speed > MIN_TAKEOFF_SPEED, sq3.c.ground_speed_next > MIN_TAKEOFF_SPEED, sq3.c.climb_rate > MIN_TAKEOFF_CLIMB_RATE), # takeoff
db.and_(sq3.c.ground_speed_prev > MAX_LANDING_SPEED, sq3.c.ground_speed < MAX_LANDING_SPEED, sq3.c.ground_speed_next < MAX_LANDING_SPEED, sq3.c.climb_rate < MAX_LANDING_SINK_RATE), # landing
)
)
.subquery()
)
# get the sender id instead of the name and consider them if the are near airports ...
sq5 = (
db.session.query(
sq4.c.timestamp, sq4.c.track, sq4.c.is_takeoff, Sender.id.label("sender_id"), Airport.id.label("airport_id"), db.func.ST_DistanceSphere(sq4.c.location, Airport.location_wkt).label("airport_distance"), Airport.country_code
)
.filter(db.and_(db.func.ST_Within(sq4.c.location, Airport.border),
db.between(Airport.style, 2, 5)))
.filter(sq4.c.name == Sender.name)
.subquery()
)
# ... and take the nearest airport
sq6 = (
db.session.query(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.sender_id, sq5.c.airport_id, sq5.c.country_code)
.distinct(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.sender_id)
.order_by(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.sender_id, sq5.c.airport_distance)
.subquery()
)
# ... add the country
takeoff_landing_query = (
db.session.query(sq6.c.timestamp, sq6.c.track, sq6.c.is_takeoff, sq6.c.sender_id, sq6.c.airport_id, Country.gid)
.join(Country, sq6.c.country_code == Country.iso2, isouter=True)
.subquery()
)
# ... and save them
ins = insert(TakeoffLanding) \
.from_select((TakeoffLanding.timestamp, TakeoffLanding.track, TakeoffLanding.is_takeoff, TakeoffLanding.sender_id, TakeoffLanding.airport_id, TakeoffLanding.country_id), takeoff_landing_query) \
.on_conflict_do_nothing(index_elements=[TakeoffLanding.timestamp, TakeoffLanding.sender_id, TakeoffLanding.airport_id])
result = db.session.execute(ins)
db.session.commit()
insert_counter = result.rowcount
finish_message = "TakeoffLandings: {} inserted".format(insert_counter)
current_app.logger.info(finish_message)
return finish_message
def update_logbook(offset_days=None):
"""Add/update logbook entries."""
current_app.logger.info("Compute logbook.")
# limit time range to given date and set window partition and window order
if offset_days:
(start, end) = date_to_timestamps(datetime.utcnow() - timedelta(days=offset_days))
else:
(start, end) = date_to_timestamps(datetime.utcnow().date())
pa = TakeoffLanding.sender_id
wo = db.and_(TakeoffLanding.sender_id, TakeoffLanding.timestamp, TakeoffLanding.airport_id)
# make a query with previous, current and next "takeoff_landing" event, so we can find complete flights
sq = (
db.session.query(
TakeoffLanding.sender_id,
db.func.lag(TakeoffLanding.sender_id).over(partition_by=pa, order_by=wo).label("sender_id_prev"),
db.func.lead(TakeoffLanding.sender_id).over(partition_by=pa, order_by=wo).label("sender_id_next"),
TakeoffLanding.timestamp,
db.func.lag(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_prev"),
db.func.lead(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_next"),
TakeoffLanding.track,
db.func.lag(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_prev"),
db.func.lead(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_next"),
TakeoffLanding.is_takeoff,
db.func.lag(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_prev"),
db.func.lead(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_next"),
TakeoffLanding.airport_id,
db.func.lag(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_prev"),
db.func.lead(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_next")
)
.subquery()
)
# find (new) starts without landing
only_starts_query = (
db.session.query(
sq.c.sender_id.label("sender_id"),
sq.c.timestamp.label("takeoff_timestamp"),
sq.c.track.label("takeoff_track"),
sq.c.airport_id.label("takeoff_airport_id")
)
.filter(sq.c.is_takeoff == db.true())
.filter(db.or_(sq.c.is_takeoff_next == db.true(), sq.c.is_takeoff_next == db.null()))
.filter(~Logbook.query.filter(db.and_(Logbook.sender_id == sq.c.sender_id, Logbook.takeoff_timestamp == sq.c.timestamp, Logbook.takeoff_airport_id == sq.c.airport_id)).exists())
)
ins = insert(Logbook).from_select(
(
Logbook.sender_id,
Logbook.takeoff_timestamp,
Logbook.takeoff_track,
Logbook.takeoff_airport_id
),
only_starts_query,
)
result = db.session.execute(ins)
current_app.logger.debug(f"Added {result.rowcount} starts")
db.session.commit()
# find (new) landings without start
only_landings_query = (
session.query(
sq.c.timestamp.label("reftime"),
sq.c.device_id.label("device_id"),
null().label("takeoff_timestamp"),
null().label("takeoff_track"),
null().label("takeoff_airport_id"),
db.session.query(
sq.c.sender_id.label("sender_id"),
sq.c.timestamp.label("landing_timestamp"),
sq.c.track.label("landing_track"),
sq.c.airport_id.label("landing_airport_id"),
)
.filter(sq.c.is_takeoff == false())
.filter(or_(sq.c.is_takeoff_prev == false(), sq.c.is_takeoff_prev == null()))
.filter(db.or_(sq.c.is_takeoff_prev == db.false(), sq.c.is_takeoff_prev == db.null()))
.filter(sq.c.is_takeoff == db.false())
.filter(~Logbook.query.filter(db.and_(Logbook.sender_id == sq.c.sender_id, Logbook.landing_timestamp == sq.c.timestamp, Logbook.landing_airport_id == sq.c.airport_id)).exists())
)
ins = insert(Logbook).from_select(
(
Logbook.sender_id,
Logbook.landing_timestamp,
Logbook.landing_track,
Logbook.landing_airport_id
),
only_landings_query,
)
result = db.session.execute(ins)
current_app.logger.debug(f"Added {result.rowcount} landings")
db.session.commit()
# find starts without landing
only_starts_query = (
session.query(
sq.c.timestamp.label("reftime"),
sq.c.device_id.label("device_id"),
# find complete flights
complete_flight_query = (
db.session.query(
sq.c.sender_id.label("sender_id"),
sq.c.timestamp.label("takeoff_timestamp"),
sq.c.track.label("takeoff_track"),
sq.c.airport_id.label("takeoff_airport_id"),
null().label("landing_timestamp"),
null().label("landing_track"),
null().label("landing_airport_id"),
sq.c.timestamp_next.label("landing_timestamp"),
sq.c.track_next.label("landing_track"),
sq.c.airport_id_next.label("landing_airport_id"),
)
.filter(sq.c.is_takeoff == true())
.filter(or_(sq.c.is_takeoff_next == true(), sq.c.is_takeoff_next == null()))
.filter(sq.c.is_takeoff == db.true())
.filter(sq.c.is_takeoff_next == db.false())
.subquery()
)
# unite all computated flights
union_query = complete_flight_query.union(only_landings_query, only_starts_query).subquery()
# if a logbook entry exist --> update it
upd = (
update(Logbook)
.where(
and_(
Logbook.device_id == union_query.c.device_id,
union_query.c.takeoff_airport_id != null(),
union_query.c.landing_airport_id != null(),
or_(
and_(Logbook.takeoff_airport_id == union_query.c.takeoff_airport_id, Logbook.takeoff_timestamp == union_query.c.takeoff_timestamp, Logbook.landing_airport_id == null()),
and_(Logbook.takeoff_airport_id == null(), Logbook.landing_airport_id == union_query.c.landing_airport_id, Logbook.landing_timestamp == union_query.c.landing_timestamp),
),
)
)
.values(
{
"reftime": union_query.c.reftime,
"takeoff_timestamp": union_query.c.takeoff_timestamp,
"takeoff_track": union_query.c.takeoff_track,
"takeoff_airport_id": union_query.c.takeoff_airport_id,
"landing_timestamp": union_query.c.landing_timestamp,
"landing_track": union_query.c.landing_track,
"landing_airport_id": union_query.c.landing_airport_id,
}
)
# insert (new) flights
new_flights_query = (
db.session.query(complete_flight_query)
.filter(~Logbook.query.filter(db.and_(Logbook.sender_id == complete_flight_query.c.sender_id, Logbook.landing_timestamp == complete_flight_query.c.landing_timestamp, Logbook.landing_airport_id == complete_flight_query.c.landing_airport_id)).exists())
.filter(~Logbook.query.filter(db.and_(Logbook.sender_id == complete_flight_query.c.sender_id, Logbook.takeoff_timestamp == complete_flight_query.c.takeoff_timestamp, Logbook.takeoff_airport_id == complete_flight_query.c.takeoff_airport_id)).exists())
)
result = session.execute(upd)
update_counter = result.rowcount
session.commit()
logger.debug("Updated logbook entries: {}".format(update_counter))
# if a logbook entry doesnt exist --> insert it
new_logbook_entries = session.query(union_query).filter(
~exists().where(
and_(
Logbook.device_id == union_query.c.device_id,
or_(
and_(Logbook.takeoff_airport_id == union_query.c.takeoff_airport_id, Logbook.takeoff_timestamp == union_query.c.takeoff_timestamp),
and_(Logbook.takeoff_airport_id == null(), union_query.c.takeoff_airport_id == null()),
),
or_(
and_(Logbook.landing_airport_id == union_query.c.landing_airport_id, Logbook.landing_timestamp == union_query.c.landing_timestamp),
and_(Logbook.landing_airport_id == null(), union_query.c.landing_airport_id == null()),
),
)
)
)
ins = insert(Logbook).from_select(
(
Logbook.reftime,
Logbook.device_id,
Logbook.sender_id,
Logbook.takeoff_timestamp,
Logbook.takeoff_track,
Logbook.takeoff_airport_id,
Logbook.landing_timestamp,
Logbook.landing_track,
Logbook.landing_airport_id,
Logbook.landing_airport_id
),
new_logbook_entries,
new_flights_query
)
result = db.session.execute(ins)
current_app.logger.debug(f"Added {result.rowcount} complete flights")
db.session.commit()
result = session.execute(ins)
insert_counter = result.rowcount
session.commit()
# update existing landing with takeoff from complete flight
upd = db.update(Logbook) \
.where(db.and_(
Logbook.sender_id == complete_flight_query.c.sender_id,
Logbook.takeoff_timestamp == db.null(),
Logbook.takeoff_airport_id == db.null(),
Logbook.landing_timestamp != db.null(),
Logbook.landing_timestamp == complete_flight_query.c.landing_timestamp,
Logbook.landing_airport_id == complete_flight_query.c.landing_airport_id
)) \
.values(takeoff_timestamp=complete_flight_query.c.takeoff_timestamp,
takeoff_track=complete_flight_query.c.takeoff_track,
takeoff_airport_id=complete_flight_query.c.takeoff_airport_id)
result = db.session.execute(upd)
current_app.logger.debug(f"Updated {result.rowcount} takeoffs to complete flights")
db.session.commit()
finish_message = "Logbook: {} inserted, {} updated".format(insert_counter, update_counter)
logger.debug(finish_message)
return finish_message
# update existing takeoff with landing from complete flight
upd = db.update(Logbook) \
.where(db.and_(
Logbook.sender_id == complete_flight_query.c.sender_id,
Logbook.takeoff_timestamp != db.null(),
Logbook.takeoff_timestamp == complete_flight_query.c.takeoff_timestamp,
Logbook.takeoff_airport_id == complete_flight_query.c.takeoff_airport_id,
Logbook.landing_timestamp == db.null(),
Logbook.landing_airport_id == db.null()
)) \
.values(landing_timestamp=complete_flight_query.c.landing_timestamp,
landing_track=complete_flight_query.c.landing_track,
landing_airport_id=complete_flight_query.c.landing_airport_id)
result = db.session.execute(upd)
current_app.logger.debug(f"Updated {result.rowcount} landings to complete flights")
db.session.commit()
return
def update_max_altitudes(session, date, logger=None):
def update_max_altitudes():
MAX_UPDATES = 60
query = """
UPDATE logbooks
SET max_altitude = sq2.max_altitude
FROM (
SELECT sq.logbook_id, MAX(sp.altitude) AS max_altitude
FROM (
SELECT
l.id AS logbook_id, s.name, l.takeoff_timestamp, l.landing_timestamp
FROM logbooks AS l
INNER JOIN senders AS s ON l.sender_id = s.id
WHERE
l.takeoff_timestamp IS NOT NULL
AND l.landing_timestamp IS NOT NULL
AND l.max_altitude IS NULL
LIMIT 1
) AS sq,
sender_positions AS sp
WHERE sp.reference_timestamp BETWEEN sq.takeoff_timestamp AND sq.landing_timestamp
AND sp.name = sq.name
GROUP BY sq.logbook_id
) AS sq2
WHERE logbooks.id = sq2.logbook_id;
"""
update_counter = 0
for _ in range(MAX_UPDATES):
result = db.session.execute(query)
db.session.commit()
return update_counter
def update_max_altitudes_orm():
"""Add max altitudes in logbook when flight is complete (takeoff and landing)."""
if logger is None:
logger = current_app.logger
logger.info("Update logbook max altitude.")
if session is None:
session = current_app.session
(start, end) = date_to_timestamps(date)
current_app.logger.info("Update logbook max altitude.")
logbook_entries = (
session.query(Logbook.id)
.filter(and_(Logbook.takeoff_timestamp != null(), Logbook.landing_timestamp != null(), Logbook.max_altitude == null()))
.filter(between(Logbook.reftime, start, end))
db.session.query(Logbook.id, Sender.name)
.filter(db.and_(Logbook.takeoff_timestamp != db.null(), Logbook.landing_timestamp != db.null(), Logbook.max_altitude == db.null()))
.filter(Logbook.sender_id == Sender.id)
.limit(1)
.subquery()
)
max_altitudes = (
session.query(Logbook.id, func.max(AircraftBeacon.altitude).label("max_altitude"))
.filter(Logbook.id == logbook_entries.c.id)
.filter(and_(AircraftBeacon.device_id == Logbook.device_id, AircraftBeacon.timestamp >= Logbook.takeoff_timestamp, AircraftBeacon.timestamp <= Logbook.landing_timestamp))
db.session.query(logbook_entries.c.id, db.func.max(SenderPosition.altitude).label("max_altitude"))
.filter(db.and_(db.between_(SenderPosition.timestamp >= Logbook.takeoff_timestamp, SenderPosition.timestamp <= Logbook.landing_timestamp), SenderPosition.name == logbook_entries.c.name))
.group_by(Logbook.id)
.subquery()
)
update_logbook = session.query(Logbook).filter(Logbook.id == max_altitudes.c.id).update({Logbook.max_altitude: max_altitudes.c.max_altitude}, synchronize_session="fetch")
update_logbooks = db.session.query(Logbook).filter(Logbook.id == max_altitudes.c.id).update({Logbook.max_altitude: max_altitudes.c.max_altitude}, synchronize_session="fetch")
session.commit()
db.session.commit()
finish_message = "Logbook (altitude): {} entries updated.".format(update_logbook)
logger.info(finish_message)
finish_message = "Logbook (altitude): {} entries updated.".format(update_logbooks)
return finish_message
if __name__ == '__main__':
from app import create_app
app = create_app()
with app.app_context():
result = update_takeoff_landings(start=datetime(2020, 11, 9, 10, 0, 0), end=datetime(2020, 11, 9, 15, 30, 0))
result = update_logbook()
result = update_max_altitudes_orm()
print(result)

104
app/collect/ognrange.py
Wyświetl plik

@ -1,104 +0,0 @@
from sqlalchemy import Date
from sqlalchemy import and_, insert, update, exists, between
from sqlalchemy.sql import func, null
from flask import current_app
from app.model import AircraftBeacon, Receiver, ReceiverCoverage
from app.utils import date_to_timestamps
def update_entries(session, date, logger=None):
"""Create receiver coverage stats for Melissas ognrange."""
if logger is None:
logger = current_app.logger
logger.info("Compute receiver coverages.")
(start, end) = date_to_timestamps(date)
# Filter aircraft beacons
sq = (
session.query(AircraftBeacon.location_mgrs_short, AircraftBeacon.receiver_name, AircraftBeacon.signal_quality, AircraftBeacon.altitude, AircraftBeacon.address)
.filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.location_mgrs_short != null(), AircraftBeacon.receiver_name != null(), AircraftBeacon.address != null()))
.subquery()
)
# ... and group them by reduced MGRS, receiver and date
sq2 = (
session.query(
sq.c.location_mgrs_short,
sq.c.receiver_name,
func.cast(date, Date).label("date"),
func.max(sq.c.signal_quality).label("max_signal_quality"),
func.min(sq.c.altitude).label("min_altitude"),
func.max(sq.c.altitude).label("max_altitude"),
func.count(sq.c.altitude).label("aircraft_beacon_count"),
func.count(func.distinct(sq.c.address)).label("device_count"),
)
.group_by(sq.c.location_mgrs_short, sq.c.receiver_name)
.subquery()
)
# Replace receiver_name with receiver_id
sq3 = (
session.query(
sq2.c.location_mgrs_short,
Receiver.id.label("receiver_id"),
sq2.c.date,
sq2.c.max_signal_quality,
sq2.c.min_altitude,
sq2.c.max_altitude,
sq2.c.aircraft_beacon_count,
sq2.c.device_count,
)
.filter(sq2.c.receiver_name == Receiver.name)
.subquery()
)
# if a receiver coverage entry exist --> update it
upd = (
update(ReceiverCoverage)
.where(and_(ReceiverCoverage.location_mgrs_short == sq3.c.location_mgrs_short, ReceiverCoverage.receiver_id == sq3.c.receiver_id, ReceiverCoverage.date == date))
.values(
{
"max_signal_quality": sq3.c.max_signal_quality,
"min_altitude": sq3.c.min_altitude,
"max_altitude": sq3.c.max_altitude,
"aircraft_beacon_count": sq3.c.aircraft_beacon_count,
"device_count": sq3.c.device_count,
}
)
)
result = session.execute(upd)
update_counter = result.rowcount
session.commit()
logger.debug("Updated receiver coverage entries: {}".format(update_counter))
# if a receiver coverage entry doesnt exist --> insert it
new_coverage_entries = session.query(sq3).filter(
~exists().where(and_(ReceiverCoverage.location_mgrs_short == sq3.c.location_mgrs_short, ReceiverCoverage.receiver_id == sq3.c.receiver_id, ReceiverCoverage.date == date))
)
ins = insert(ReceiverCoverage).from_select(
(
ReceiverCoverage.location_mgrs_short,
ReceiverCoverage.receiver_id,
ReceiverCoverage.date,
ReceiverCoverage.max_signal_quality,
ReceiverCoverage.min_altitude,
ReceiverCoverage.max_altitude,
ReceiverCoverage.aircraft_beacon_count,
ReceiverCoverage.device_count,
),
new_coverage_entries,
)
result = session.execute(ins)
insert_counter = result.rowcount
session.commit()
finish_message = "ReceiverCoverage: {} inserted, {} updated".format(insert_counter, update_counter)
logger.debug(finish_message)
return finish_message

450
app/collect/stats.py
Wyświetl plik

@ -1,450 +0,0 @@
from flask import current_app
from sqlalchemy import insert, distinct, between, literal
from sqlalchemy.sql import null, and_, func, or_, update
from sqlalchemy.sql.expression import case
from app.model import AircraftBeacon, DeviceStats, Country, CountryStats, ReceiverStats, ReceiverBeacon, RelationStats, Receiver, Device
from app.utils import date_to_timestamps
# 40dB@10km is enough for 640km
MAX_PLAUSIBLE_QUALITY = 40
def create_device_stats(session, date, logger=None):
"""Add/update device stats."""
if logger is None:
logger = current_app.logger
(start, end) = date_to_timestamps(date)
# First kill the stats for the selected date
deleted_counter = session.query(DeviceStats).filter(DeviceStats.date == date).delete()
# Since "distinct count" does not work in window functions we need a work-around for receiver counting
sq = (
session.query(AircraftBeacon, func.dense_rank().over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.receiver_id).label("dr"))
.filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.device_id != null()))
.filter(or_(AircraftBeacon.error_count == 0, AircraftBeacon.error_count == null()))
.subquery()
)
# Calculate stats, firstseen, lastseen and last values != NULL
device_stats = session.query(
distinct(sq.c.device_id).label("device_id"),
literal(date).label("date"),
func.max(sq.c.dr).over(partition_by=sq.c.device_id).label("receiver_count"),
func.max(sq.c.altitude).over(partition_by=sq.c.device_id).label("max_altitude"),
func.count(sq.c.device_id).over(partition_by=sq.c.device_id).label("aircraft_beacon_count"),
func.first_value(sq.c.name).over(partition_by=sq.c.device_id, order_by=case([(sq.c.name == null(), None)], else_=sq.c.timestamp).asc().nullslast()).label("name"),
func.first_value(sq.c.timestamp).over(partition_by=sq.c.device_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).asc().nullslast()).label("firstseen"),
func.first_value(sq.c.timestamp).over(partition_by=sq.c.device_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("lastseen"),
func.first_value(sq.c.aircraft_type).over(partition_by=sq.c.device_id, order_by=case([(sq.c.aircraft_type == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("aircraft_type"),
func.first_value(sq.c.stealth).over(partition_by=sq.c.device_id, order_by=case([(sq.c.stealth == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("stealth"),
func.first_value(sq.c.software_version)
.over(partition_by=sq.c.device_id, order_by=case([(sq.c.software_version == null(), None)], else_=sq.c.timestamp).desc().nullslast())
.label("software_version"),
func.first_value(sq.c.hardware_version)
.over(partition_by=sq.c.device_id, order_by=case([(sq.c.hardware_version == null(), None)], else_=sq.c.timestamp).desc().nullslast())
.label("hardware_version"),
func.first_value(sq.c.real_address).over(partition_by=sq.c.device_id, order_by=case([(sq.c.real_address == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("real_address"),
).subquery()
# And insert them
ins = insert(DeviceStats).from_select(
[
DeviceStats.device_id,
DeviceStats.date,
DeviceStats.receiver_count,
DeviceStats.max_altitude,
DeviceStats.aircraft_beacon_count,
DeviceStats.name,
DeviceStats.firstseen,
DeviceStats.lastseen,
DeviceStats.aircraft_type,
DeviceStats.stealth,
DeviceStats.software_version,
DeviceStats.hardware_version,
DeviceStats.real_address,
],
device_stats,
)
res = session.execute(ins)
insert_counter = res.rowcount
session.commit()
logger.debug("DeviceStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter))
return "DeviceStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter)
def create_receiver_stats(session, date, logger=None):
"""Add/update receiver stats."""
if logger is None:
logger = current_app.logger
(start, end) = date_to_timestamps(date)
# First kill the stats for the selected date
deleted_counter = session.query(ReceiverStats).filter(ReceiverStats.date == date).delete()
# Select one day
sq = session.query(ReceiverBeacon).filter(between(ReceiverBeacon.timestamp, start, end)).subquery()
# Calculate stats, firstseen, lastseen and last values != NULL
receiver_stats = session.query(
distinct(sq.c.receiver_id).label("receiver_id"),
literal(date).label("date"),
func.first_value(sq.c.timestamp).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).asc().nullslast()).label("firstseen"),
func.first_value(sq.c.timestamp).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("lastseen"),
func.first_value(sq.c.location).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.location == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("location_wkt"),
func.first_value(sq.c.altitude).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.altitude == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("altitude"),
func.first_value(sq.c.version).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.version == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("version"),
func.first_value(sq.c.platform).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.platform == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("platform"),
).subquery()
# And insert them
ins = insert(ReceiverStats).from_select(
[
ReceiverStats.receiver_id,
ReceiverStats.date,
ReceiverStats.firstseen,
ReceiverStats.lastseen,
ReceiverStats.location_wkt,
ReceiverStats.altitude,
ReceiverStats.version,
ReceiverStats.platform,
],
receiver_stats,
)
res = session.execute(ins)
insert_counter = res.rowcount
session.commit()
logger.warn("ReceiverStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter))
# Update aircraft_beacon_count, aircraft_count and max_distance
aircraft_beacon_stats = (
session.query(
AircraftBeacon.receiver_id,
func.count(AircraftBeacon.timestamp).label("aircraft_beacon_count"),
func.count(func.distinct(AircraftBeacon.device_id)).label("aircraft_count"),
func.max(AircraftBeacon.distance).label("max_distance"),
)
.filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.error_count == 0, AircraftBeacon.quality <= MAX_PLAUSIBLE_QUALITY, AircraftBeacon.relay == null()))
.group_by(AircraftBeacon.receiver_id)
.subquery()
)
upd = (
update(ReceiverStats)
.where(and_(ReceiverStats.date == date, ReceiverStats.receiver_id == aircraft_beacon_stats.c.receiver_id))
.values(
{"aircraft_beacon_count": aircraft_beacon_stats.c.aircraft_beacon_count, "aircraft_count": aircraft_beacon_stats.c.aircraft_count, "max_distance": aircraft_beacon_stats.c.max_distance}
)
)
result = session.execute(upd)
update_counter = result.rowcount
session.commit()
logger.warn("Updated {} ReceiverStats".format(update_counter))
return "ReceiverStats for {}: {} deleted, {} inserted, {} updated".format(date, deleted_counter, insert_counter, update_counter)
def create_country_stats(session, date, logger=None):
if logger is None:
logger = current_app.logger
(start, end) = date_to_timestamps(date)
# First kill the stats for the selected date
deleted_counter = session.query(CountryStats).filter(CountryStats.date == date).delete()
country_stats = (
session.query(literal(date), Country.gid, func.count(AircraftBeacon.timestamp).label("aircraft_beacon_count"), func.count(func.distinct(AircraftBeacon.receiver_id)).label("device_count"))
.filter(between(AircraftBeacon.timestamp, start, end))
.filter(func.st_contains(Country.geom, AircraftBeacon.location))
.group_by(Country.gid)
.subquery()
)
# And insert them
ins = insert(CountryStats).from_select([CountryStats.date, CountryStats.country_id, CountryStats.aircraft_beacon_count, CountryStats.device_count], country_stats)
res = session.execute(ins)
insert_counter = res.rowcount
session.commit()
def update_device_stats_jumps(session, date, logger=None):
"""Update device stats jumps."""
if logger is None:
logger = current_app.logger
(start, end) = date_to_timestamps(date)
# speed limits in m/s (values above indicates a unplausible position / jump)
max_horizontal_speed = 1000
max_vertical_speed = 100
max_jumps = 10 # threshold for an 'ambiguous' device
# find consecutive positions for a device
sq = (
session.query(
AircraftBeacon.device_id,
AircraftBeacon.timestamp,
func.lead(AircraftBeacon.timestamp).over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.timestamp).label("timestamp_next"),
AircraftBeacon.location_wkt,
func.lead(AircraftBeacon.location_wkt).over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.timestamp).label("location_next"),
AircraftBeacon.altitude,
func.lead(AircraftBeacon.altitude).over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.timestamp).label("altitude_next"),
)
.filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.error_count == 0))
.subquery()
)
# calc vertial and horizontal speed between points
sq2 = (
session.query(
sq.c.device_id,
(func.st_distancesphere(sq.c.location_next, sq.c.location) / (func.extract("epoch", sq.c.timestamp_next) - func.extract("epoch", sq.c.timestamp))).label("horizontal_speed"),
((sq.c.altitude_next - sq.c.altitude) / (func.extract("epoch", sq.c.timestamp_next) - func.extract("epoch", sq.c.timestamp))).label("vertical_speed"),
)
.filter(and_(sq.c.timestamp != null(), sq.c.timestamp_next != null(), sq.c.timestamp < sq.c.timestamp_next))
.subquery()
)
# ... and find and count 'jumps'
sq3 = (
session.query(sq2.c.device_id, func.sum(case([(or_(func.abs(sq2.c.horizontal_speed) > max_horizontal_speed, func.abs(sq2.c.vertical_speed) > max_vertical_speed), 1)], else_=0)).label("jumps"))
.group_by(sq2.c.device_id)
.subquery()
)
upd = update(DeviceStats).where(and_(DeviceStats.date == date, DeviceStats.device_id == sq3.c.device_id)).values({"ambiguous": sq3.c.jumps > max_jumps, "jumps": sq3.c.jumps})
result = session.execute(upd)
update_counter = result.rowcount
session.commit()
logger.warn("Updated {} DeviceStats jumps".format(update_counter))
return "DeviceStats jumps for {}: {} updated".format(date, update_counter)
def create_relation_stats(session, date, logger=None):
"""Add/update relation stats."""
if logger is None:
logger = current_app.logger
(start, end) = date_to_timestamps(date)
# First kill the stats for the selected date
deleted_counter = session.query(RelationStats).filter(RelationStats.date == date).delete()
# Calculate stats for selected day
relation_stats = (
session.query(literal(date), AircraftBeacon.device_id, AircraftBeacon.receiver_id, func.max(AircraftBeacon.quality), func.count(AircraftBeacon.timestamp))
.filter(
and_(
between(AircraftBeacon.timestamp, start, end),
AircraftBeacon.distance > 1000,
AircraftBeacon.error_count == 0,
AircraftBeacon.quality <= MAX_PLAUSIBLE_QUALITY,
AircraftBeacon.ground_speed > 10,
)
)
.group_by(literal(date), AircraftBeacon.device_id, AircraftBeacon.receiver_id)
.subquery()
)
# And insert them
ins = insert(RelationStats).from_select([RelationStats.date, RelationStats.device_id, RelationStats.receiver_id, RelationStats.quality, RelationStats.beacon_count], relation_stats)
res = session.execute(ins)
insert_counter = res.rowcount
session.commit()
logger.warn("RelationStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter))
return "RelationStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter)
def update_qualities(session, date, logger=None):
"""Calculate relative qualities of receivers and devices."""
if logger is None:
logger = current_app.logger
# Calculate avg quality of devices
dev_sq = session.query(RelationStats.device_id, func.avg(RelationStats.quality).label("quality")).filter(RelationStats.date == date).group_by(RelationStats.device_id).subquery()
dev_upd = update(DeviceStats).where(and_(DeviceStats.date == date, DeviceStats.device_id == dev_sq.c.device_id)).values({"quality": dev_sq.c.quality})
dev_result = session.execute(dev_upd)
dev_update_counter = dev_result.rowcount
session.commit()
logger.warn("Updated {} DeviceStats: quality".format(dev_update_counter))
# Calculate avg quality of receivers
rec_sq = session.query(RelationStats.receiver_id, func.avg(RelationStats.quality).label("quality")).filter(RelationStats.date == date).group_by(RelationStats.receiver_id).subquery()
rec_upd = update(ReceiverStats).where(and_(ReceiverStats.date == date, ReceiverStats.receiver_id == rec_sq.c.receiver_id)).values({"quality": rec_sq.c.quality})
rec_result = session.execute(rec_upd)
rec_update_counter = rec_result.rowcount
session.commit()
logger.warn("Updated {} ReceiverStats: quality".format(rec_update_counter))
# Calculate quality_offset of devices
dev_sq = (
session.query(
RelationStats.device_id, (func.sum(RelationStats.beacon_count * (RelationStats.quality - ReceiverStats.quality)) / (func.sum(RelationStats.beacon_count))).label("quality_offset")
)
.filter(RelationStats.date == date)
.filter(and_(RelationStats.receiver_id == ReceiverStats.receiver_id, RelationStats.date == ReceiverStats.date))
.group_by(RelationStats.device_id)
.subquery()
)
dev_upd = update(DeviceStats).where(and_(DeviceStats.date == date, DeviceStats.device_id == dev_sq.c.device_id)).values({"quality_offset": dev_sq.c.quality_offset})
dev_result = session.execute(dev_upd)
dev_update_counter = dev_result.rowcount
session.commit()
logger.warn("Updated {} DeviceStats: quality_offset".format(dev_update_counter))
# Calculate quality_offset of receivers
rec_sq = (
session.query(
RelationStats.receiver_id, (func.sum(RelationStats.beacon_count * (RelationStats.quality - DeviceStats.quality)) / (func.sum(RelationStats.beacon_count))).label("quality_offset")
)
.filter(RelationStats.date == date)
.filter(and_(RelationStats.device_id == DeviceStats.device_id, RelationStats.date == DeviceStats.date))
.group_by(RelationStats.receiver_id)
.subquery()
)
rec_upd = update(ReceiverStats).where(and_(ReceiverStats.date == date, ReceiverStats.receiver_id == rec_sq.c.receiver_id)).values({"quality_offset": rec_sq.c.quality_offset})
rec_result = session.execute(rec_upd)
rec_update_counter = rec_result.rowcount
session.commit()
logger.warn("Updated {} ReceiverStats: quality_offset".format(rec_update_counter))
return "Updated {} DeviceStats and {} ReceiverStats".format(dev_update_counter, rec_update_counter)
def update_receivers(session, logger=None):
"""Update receivers with stats."""
if logger is None:
logger = current_app.logger
receiver_stats = (
session.query(
distinct(ReceiverStats.receiver_id).label("receiver_id"),
func.first_value(ReceiverStats.firstseen)
.over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.firstseen == null(), None)], else_=ReceiverStats.date).asc().nullslast())
.label("firstseen"),
func.first_value(ReceiverStats.lastseen)
.over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.lastseen == null(), None)], else_=ReceiverStats.date).desc().nullslast())
.label("lastseen"),
func.first_value(ReceiverStats.location_wkt)
.over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.location_wkt == null(), None)], else_=ReceiverStats.date).desc().nullslast())
.label("location_wkt"),
func.first_value(ReceiverStats.altitude)
.over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.altitude == null(), None)], else_=ReceiverStats.date).desc().nullslast())
.label("altitude"),
func.first_value(ReceiverStats.version)
.over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.version == null(), None)], else_=ReceiverStats.date).desc().nullslast())
.label("version"),
func.first_value(ReceiverStats.platform)
.over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.platform == null(), None)], else_=ReceiverStats.date).desc().nullslast())
.label("platform"),
)
.order_by(ReceiverStats.receiver_id)
.subquery()
)
upd = (
update(Receiver)
.where(and_(Receiver.id == receiver_stats.c.receiver_id))
.values(
{
"firstseen": receiver_stats.c.firstseen,
"lastseen": receiver_stats.c.lastseen,
"location": receiver_stats.c.location_wkt,
"altitude": receiver_stats.c.altitude,
"version": receiver_stats.c.version,
"platform": receiver_stats.c.platform,
}
)
)
result = session.execute(upd)
update_counter = result.rowcount
session.commit()
logger.warn("Updated {} Receivers".format(update_counter))
return "Updated {} Receivers".format(update_counter)
def update_devices(session, logger=None):
"""Update devices with stats."""
if logger is None:
logger = current_app.logger
device_stats = (
session.query(
distinct(DeviceStats.device_id).label("device_id"),
func.first_value(DeviceStats.name).over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.name == null(), None)], else_=DeviceStats.date).desc().nullslast()).label("name"),
func.first_value(DeviceStats.firstseen)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.firstseen == null(), None)], else_=DeviceStats.date).asc().nullslast())
.label("firstseen"),
func.max(DeviceStats.lastseen)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.lastseen == null(), None)], else_=DeviceStats.date).desc().nullslast())
.label("lastseen"),
func.first_value(DeviceStats.aircraft_type)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.aircraft_type == null(), None)], else_=DeviceStats.date).desc().nullslast())
.label("aircraft_type"),
func.first_value(DeviceStats.stealth)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.stealth == null(), None)], else_=DeviceStats.date).desc().nullslast())
.label("stealth"),
func.first_value(DeviceStats.software_version)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.software_version == null(), None)], else_=DeviceStats.date).desc().nullslast())
.label("software_version"),
func.first_value(DeviceStats.hardware_version)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.hardware_version == null(), None)], else_=DeviceStats.date).desc().nullslast())
.label("hardware_version"),
func.first_value(DeviceStats.real_address)
.over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.real_address == null(), None)], else_=DeviceStats.date).desc().nullslast())
.label("real_address"),
)
.order_by(DeviceStats.device_id)
.subquery()
)
upd = (
update(Device)
.where(and_(Device.id == device_stats.c.device_id))
.values(
{
"name": device_stats.c.name,
"firstseen": device_stats.c.firstseen,
"lastseen": device_stats.c.lastseen,
"aircraft_type": device_stats.c.aircraft_type,
"stealth": device_stats.c.stealth,
"software_version": device_stats.c.software_version,
"hardware_version": device_stats.c.hardware_version,
"real_address": device_stats.c.real_address,
}
)
)
result = session.execute(upd)
update_counter = result.rowcount
session.commit()
logger.warn("Updated {} Devices".format(update_counter))
return "Updated {} Devices".format(update_counter)

145
app/collect/takeoff_landings.py
Wyświetl plik

@ -1,145 +0,0 @@
from datetime import timedelta
from flask import current_app
from sqlalchemy import and_, or_, insert, between, exists
from sqlalchemy.sql import func, null
from sqlalchemy.sql.expression import case
from app.model import AircraftBeacon, Device, TakeoffLanding, Airport
def update_entries(session, start, end, logger=None):
"""Compute takeoffs and landings."""
if logger is None:
logger = current_app.logger
logger.info("Compute takeoffs and landings.")
# considered time interval should not exceed a complete day
if end - start > timedelta(days=1):
abort_message = "TakeoffLanding: timeinterval start='{}' and end='{}' is too big.".format(start, end)
logger.warn(abort_message)
return abort_message
# check if we have any airport
airports_query = session.query(Airport).limit(1)
if not airports_query.all():
abort_message = "TakeoffLanding: Cannot calculate takeoff and landings without any airport! Please import airports first."
logger.warn(abort_message)
return abort_message
# takeoff / landing detection is based on 3 consecutive points all below a certain altitude AGL
takeoff_speed = 55 # takeoff detection: 1st point below, 2nd and 3rd above this limit
landing_speed = 40 # landing detection: 1st point above, 2nd and 3rd below this limit
min_takeoff_climb_rate = -5 # takeoff detection: glider should not sink too much
max_landing_climb_rate = 5 # landing detection: glider should not climb too much
duration = 100 # the points must not exceed this duration
radius = 5000 # the points must not exceed this radius around the 2nd point
max_agl = 200 # takeoff / landing must not exceed this altitude AGL
# get beacons for selected time range, one per address and timestamp
sq = (
session.query(AircraftBeacon)
.distinct(AircraftBeacon.address, AircraftBeacon.timestamp)
.order_by(AircraftBeacon.address, AircraftBeacon.timestamp, AircraftBeacon.error_count)
.filter(AircraftBeacon.agl < max_agl)
.filter(between(AircraftBeacon.timestamp, start, end))
.subquery()
)
# make a query with current, previous and next position
sq2 = session.query(
sq.c.address,
func.lag(sq.c.address).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("address_prev"),
func.lead(sq.c.address).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("address_next"),
sq.c.timestamp,
func.lag(sq.c.timestamp).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("timestamp_prev"),
func.lead(sq.c.timestamp).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("timestamp_next"),
sq.c.location,
func.lag(sq.c.location).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("location_wkt_prev"),
func.lead(sq.c.location).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("location_wkt_next"),
sq.c.track,
func.lag(sq.c.track).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("track_prev"),
func.lead(sq.c.track).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("track_next"),
sq.c.ground_speed,
func.lag(sq.c.ground_speed).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("ground_speed_prev"),
func.lead(sq.c.ground_speed).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("ground_speed_next"),
sq.c.altitude,
func.lag(sq.c.altitude).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("altitude_prev"),
func.lead(sq.c.altitude).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("altitude_next"),
sq.c.climb_rate,
func.lag(sq.c.climb_rate).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("climb_rate_prev"),
func.lead(sq.c.climb_rate).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("climb_rate_next"),
).subquery()
# consider only positions with predecessor and successor and limit distance and duration between points
sq3 = (
session.query(sq2)
.filter(and_(sq2.c.address_prev != null(), sq2.c.address_next != null()))
.filter(and_(func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_prev) < radius, func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_next) < radius))
.filter(sq2.c.timestamp_next - sq2.c.timestamp_prev < timedelta(seconds=duration))
.subquery()
)
# find possible takeoffs and landings
sq4 = (
session.query(
sq3.c.timestamp,
case(
[
(sq3.c.ground_speed > takeoff_speed, sq3.c.location_wkt_prev), # on takeoff we take the location from the previous fix because it is nearer to the airport
(sq3.c.ground_speed <= takeoff_speed, sq3.c.location),
]
).label("location"),
case([(sq3.c.ground_speed > landing_speed, sq3.c.track), (sq3.c.ground_speed <= landing_speed, sq3.c.track_prev)]).label(
"track"
), # on landing we take the track from the previous fix because gliders tend to leave the runway quickly
sq3.c.ground_speed,
sq3.c.altitude,
case([(sq3.c.ground_speed > takeoff_speed, True), (sq3.c.ground_speed < landing_speed, False)]).label("is_takeoff"),
sq3.c.address,
)
.filter(
or_(
and_(sq3.c.ground_speed_prev < takeoff_speed, sq3.c.ground_speed > takeoff_speed, sq3.c.ground_speed_next > takeoff_speed, sq3.c.climb_rate > min_takeoff_climb_rate), # takeoff
and_(sq3.c.ground_speed_prev > landing_speed, sq3.c.ground_speed < landing_speed, sq3.c.ground_speed_next < landing_speed, sq3.c.climb_rate < max_landing_climb_rate), # landing
)
)
.subquery()
)
# get the device id instead of the address and consider them if the are near airports ...
sq5 = (
session.query(
sq4.c.timestamp, sq4.c.track, sq4.c.is_takeoff, Device.id.label("device_id"), Airport.id.label("airport_id"), func.ST_DistanceSphere(sq4.c.location, Airport.location_wkt).label("airport_distance")
)
.filter(and_(sq4.c.address == Device.address,
func.ST_Within(sq4.c.location, Airport.border),
between(Airport.style, 2, 5)))
.subquery()
)
# ... and take the nearest airport
sq6 = (
session.query(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.device_id, sq5.c.airport_id)
.distinct(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.device_id)
.order_by(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.device_id, sq5.c.airport_distance)
.subquery()
)
# consider them only if they are not already existing in db
takeoff_landing_query = session.query(sq6).filter(
~exists().where(and_(TakeoffLanding.timestamp == sq6.c.timestamp, TakeoffLanding.device_id == sq6.c.device_id, TakeoffLanding.airport_id == sq6.c.airport_id))
)
# ... and save them
ins = insert(TakeoffLanding).from_select((TakeoffLanding.timestamp, TakeoffLanding.track, TakeoffLanding.is_takeoff, TakeoffLanding.device_id, TakeoffLanding.airport_id), takeoff_landing_query)
result = session.execute(ins)
session.commit()
insert_counter = result.rowcount
finish_message = "TakeoffLandings: {} inserted".format(insert_counter)
logger.info(finish_message)
return finish_message

167
app/collect/timescaledb_views.py 100644
Wyświetl plik

@ -0,0 +1,167 @@
from app import db
from app.utils import get_sql_trustworthy
SQL_TRUSTWORTHY = get_sql_trustworthy(source_table_alias='sp')
def create_views():
db.session.execute("""
DROP VIEW IF EXISTS receiver_ranking CASCADE;
CREATE VIEW receiver_ranking AS
SELECT
r.name AS receiver_name,
r.id AS receiver_id,
MAX(rs.max_distance) AS max_distance,
SUM(rs.max_normalized_quality * rs.messages_count) / SUM(rs.messages_count) AS max_normalized_quality,
SUM(rs.messages_count) AS messages_count,
COUNT(DISTINCT rs.sender_id) AS senders_count,
COUNT(DISTINCT rs.location_mgrs_short) AS coverage_count
FROM coverage_statistics AS rs
INNER JOIN receivers AS r ON rs.receiver_id = r.id
WHERE rs.date = NOW()::date AND rs.is_trustworthy IS TRUE AND rs.max_distance IS NOT NULL
GROUP BY rs.date, r.name, r.id
ORDER BY max_distance DESC;
""")
db.session.execute("""
DROP VIEW IF EXISTS sender_ranking CASCADE;
CREATE VIEW sender_ranking AS
SELECT
s.name,
s.id AS sender_id,
MAX(rs.max_distance) AS max_distance,
SUM(rs.max_normalized_quality * rs.messages_count) / SUM(rs.messages_count) AS max_normalized_quality,
SUM(rs.messages_count) AS messages_count,
COUNT(DISTINCT rs.receiver_id) AS receivers_count,
COUNT(DISTINCT rs.location_mgrs_short) AS coverage_count
FROM coverage_statistics AS rs
INNER JOIN senders AS s ON rs.sender_id = s.id
WHERE rs.date = NOW()::date AND rs.is_trustworthy IS TRUE AND rs.max_distance IS NOT NULL
GROUP BY rs.date, s.name, s.id
ORDER BY max_distance DESC;
""")
db.session.commit()
def create_timescaledb_views():
# 1. Since the reference_timestamps are strictly increasing we can set
# the parameter 'refresh_lag' to a very short time so the materialization
# starts right after the bucket is finished
# 2. The feature realtime aggregation from TimescaleDB is quite time consuming.
# So we set materialized_only=true
# --- Sender statistics ---
# These stats will be used in the daily ranking, so we make the bucket < 1d
db.session.execute(f"""
DROP VIEW IF EXISTS sender_stats_1h CASCADE;
CREATE VIEW sender_stats_1h
WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='5 minutes') AS
SELECT
time_bucket(INTERVAL '1 hour', sp.reference_timestamp) AS bucket,
sp.name,
({SQL_TRUSTWORTHY}) AS is_trustworthy,
COUNT(sp.*) AS beacon_count,
MAX(sp.distance) AS max_distance,
MIN(sp.altitude) AS min_altitude,
MAX(sp.altitude) AS max_altitude
FROM sender_positions AS sp
GROUP BY bucket, sp.name, is_trustworthy;
""")
# ... and just for curiosity also bucket = 1d
db.session.execute(f"""
DROP VIEW IF EXISTS sender_stats_1d CASCADE;
CREATE VIEW sender_stats_1d
WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='1 hour') AS
SELECT
time_bucket(INTERVAL '1 day', sp.reference_timestamp) AS bucket,
sp.name,
({SQL_TRUSTWORTHY}) AS is_trustworthy,
COUNT(sp.*) AS beacon_count,
MAX(sp.distance) AS max_distance,
MIN(sp.altitude) AS min_altitude,
MAX(sp.altitude) AS max_altitude
FROM sender_positions AS sp
GROUP BY bucket, sp.name, is_trustworthy;
""")
# --- Receiver statistics ---
# These stats will be used in the daily ranking, so we make the bucket < 1d
db.session.execute(f"""
DROP VIEW IF EXISTS receiver_stats_1h CASCADE;
CREATE VIEW receiver_stats_1h
WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='5 minutes') AS
SELECT
time_bucket(INTERVAL '1 hour', sp.reference_timestamp) AS bucket,
sp.receiver_name,
({SQL_TRUSTWORTHY}) AS is_trustworthy,
COUNT(sp.*) AS beacon_count,
MAX(sp.distance) AS max_distance,
MIN(sp.altitude) AS min_altitude,
MAX(sp.altitude) AS max_altitude
FROM sender_positions AS sp
GROUP BY bucket, sp.receiver_name, is_trustworthy;
""")
# ... and just for curiosity also bucket = 1d
db.session.execute(f"""
DROP VIEW IF EXISTS receiver_stats_1d CASCADE;
CREATE VIEW receiver_stats_1d
WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='1 hour') AS
SELECT
time_bucket(INTERVAL '1 day', sp.reference_timestamp) AS bucket,
sp.receiver_name,
({SQL_TRUSTWORTHY}) AS is_trustworthy,
COUNT(sp.*) AS beacon_count,
MAX(sp.distance) AS max_distance,
MIN(sp.altitude) AS min_altitude,
MAX(sp.altitude) AS max_altitude
FROM sender_positions AS sp
GROUP BY bucket, sp.receiver_name, is_trustworthy;
""")
# --- Relation statistics (sender <-> receiver) ---
# these stats will be used on a >= 1d basis, so we make the bucket = 1d
db.session.execute(f"""
DROP VIEW IF EXISTS relation_stats_1d CASCADE;
CREATE VIEW relation_stats_1d
WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='1 hour') AS
SELECT
time_bucket(INTERVAL '1 day', sp.reference_timestamp) AS bucket,
sp.name,
sp.receiver_name,
({SQL_TRUSTWORTHY}) AS is_trustworthy,
COUNT(sp.*) AS beacon_count,
MAX(sp.normalized_quality) AS max_normalized_quality,
MAX(sp.distance) AS max_distance
FROM sender_positions AS sp
GROUP BY bucket, sp.name, sp.receiver_name, is_trustworthy;
""")
db.session.commit()
"""
class MyView(db.Model):
__table__ = db.Table(
'device_stats', db.metadata,
db.Column('bucket', db.DateTime, primary_key=True),
db.Column('name', db.String, primary_key=True),
db.Column('beacon_count', db.Integer),
autoload=True,
autoload_with=db.engine
)
"""

2
app/commands/__init__.py
Wyświetl plik

@ -3,7 +3,6 @@ from .export import user_cli as export_cli
from .flights import user_cli as flights_cli
from .gateway import user_cli as gateway_cli
from .logbook import user_cli as logbook_cli
from .stats import user_cli as stats_cli
def register(app):
@ -12,4 +11,3 @@ def register(app):
app.cli.add_command(flights_cli)
app.cli.add_command(gateway_cli)
app.cli.add_command(logbook_cli)
app.cli.add_command(stats_cli)

54
app/commands/database.py
Wyświetl plik

@ -5,9 +5,10 @@ import click
from datetime import datetime
from sqlalchemy.sql import func
from app.collect.database import update_device_infos, update_country_code
from app.model import AircraftBeacon, DeviceInfoOrigin
from app.collect.database import update_device_infos
from app.model import SenderPosition, SenderInfoOrigin
from app.utils import get_airports, get_days
from app.collect.timescaledb_views import create_timescaledb_views, create_views
from app import db
@ -22,7 +23,7 @@ def get_database_days(start, end):
"""Returns the first and the last day in aircraft_beacons table."""
if start is None and end is None:
days_from_db = db.session.query(func.min(AircraftBeacon.timestamp).label("first_day"), func.max(AircraftBeacon.timestamp).label("last_day")).one()
days_from_db = db.session.query(func.min(SenderPosition.timestamp).label("first_day"), func.max(SenderPosition.timestamp).label("last_day")).one()
start = days_from_db[0].date()
end = days_from_db[1].date()
else:
@ -42,29 +43,26 @@ def info():
@user_cli.command("init")
def init():
"""Initialize the database."""
"""Initialize the database (with PostGIS and TimescaleDB extensions)."""
from alembic.config import Config
from alembic import command
# Create PostGIS and PostGIS extensions
db.session.execute("CREATE EXTENSION IF NOT EXISTS postgis;")
db.session.execute("CREATE EXTENSION IF NOT EXISTS btree_gist;")
db.session.execute("CREATE EXTENSION IF NOT EXISTS timescaledb;")
db.session.commit()
# Create Scheme
db.create_all()
print("Done.")
@user_cli.command("init_timescaledb")
def init_timescaledb():
"""Initialize TimescaleDB features."""
db.session.execute("CREATE EXTENSION IF NOT EXISTS timescaledb;")
db.session.execute("SELECT create_hypertable('aircraft_beacons', 'timestamp', chunk_time_interval => interval '6 hours', if_not_exists => TRUE);")
db.session.execute("SELECT create_hypertable('receiver_beacons', 'timestamp', chunk_time_interval => interval '6 hours', if_not_exists => TRUE);")
# Change (sender|receiver)_positions to TimescaleDB table
db.session.execute("SELECT create_hypertable('sender_positions', 'reference_timestamp', chunk_time_interval => interval '3 hours', if_not_exists => TRUE);")
db.session.execute("SELECT create_hypertable('receiver_positions', 'reference_timestamp', chunk_time_interval => interval '1 day', if_not_exists => TRUE);")
db.session.commit()
print("Done.")
print("Initialized the database (with PostGIS and TimescaleDB extensions).")
@user_cli.command("drop")
@ -83,7 +81,7 @@ def import_ddb():
"""Import registered devices from the DDB."""
print("Import registered devices fom the DDB...")
counter = update_device_infos(db.session, DeviceInfoOrigin.OGN_DDB)
counter = update_device_infos(SenderInfoOrigin.OGN_DDB)
print("Imported %i devices." % counter)
@ -93,7 +91,7 @@ def import_file(path="tests/custom_ddb.txt"):
"""Import registered devices from a local file."""
print("Import registered devices from '{}'...".format(path))
counter = update_device_infos(db.session, DeviceInfoOrigin.USER_DEFINED, path=path)
counter = update_device_infos(SenderInfoOrigin.USER_DEFINED, path=path)
print("Imported %i devices." % counter)
@ -103,7 +101,7 @@ def import_flarmnet(path=None):
"""Import registered devices from a local file."""
print("Import registered devices from '{}'...".format("internet" if path is None else path))
counter = update_device_infos(db.session, DeviceInfoOrigin.FLARMNET, path=path)
counter = update_device_infos(SenderInfoOrigin.FLARMNET, path=path)
print("Imported %i devices." % counter)
@ -116,13 +114,23 @@ def import_airports(path="tests/SeeYou.cup"):
airports = get_airports(path)
db.session.bulk_save_objects(airports)
db.session.commit()
db.session.execute("UPDATE airports SET border = ST_Expand(location, 0.05)")
# TODO: SRID 4087 ist nicht korrekt, aber spherical mercator 3857 wirft hier Fehler
db.session.execute("UPDATE airports AS a SET border = ST_Transform(ST_Buffer(ST_Transform(location, 4087), 1.5 * GREATEST(500, a.runway_length)), 4326);")
db.session.commit()
print("Imported {} airports.".format(len(airports)))
@user_cli.command("update_country_codes")
def update_country_codes():
"""Update country codes of all receivers."""
@user_cli.command("create_timescaledb_views")
def cmd_create_timescaledb_views():
"""Create TimescaleDB views."""
update_country_code(session=db.session)
create_timescaledb_views()
print("Done")
@user_cli.command("create_views")
def cmd_create_views():
"""Create views."""
create_views()
print("Done")

125
app/commands/export.py
Wyświetl plik

@ -4,15 +4,89 @@ import click
import datetime
import re
import csv
import os
from sqlalchemy.orm.exc import NoResultFound
from aerofiles.igc import Writer
from app.model import AircraftBeacon, Device
from app.model import SenderPosition, Sender
from app import db
user_cli = AppGroup("export")
user_cli.help = "Export data in several file formats."
@user_cli.command("debug_sql")
@click.argument("start")
@click.argument("end")
@click.argument("name")
def debug_sql(start, end, name):
"""Export data (sender_positions and receivers) as sql for debugging (and/or creating test cases)."""
# First: get all the positions (and the receiver names for later)
sql_sender_positions = f"""
SELECT reference_timestamp, name, receiver_name, timestamp, location, track, ground_speed, altitude, aircraft_type, climb_rate, turn_rate, distance, bearing, agl
FROM sender_positions
WHERE reference_timestamp BETWEEN '{start}' AND '{end}' AND name = '{name}'
ORDER BY reference_timestamp;
"""
receiver_names = []
sender_position_values = []
results = db.session.execute(sql_sender_positions)
for row in results:
if row[2] not in receiver_names:
receiver_names.append("'" + row[2] + "'")
row = [f"'{r}'" if r else "DEFAULT" for r in row]
sender_position_values.append(f"({','.join(row)})")
# Second: get the receivers
sql_receivers = f"""
SELECT name, location
FROM receivers
WHERE name IN ({','.join(receiver_names)});
"""
receiver_values = []
results = db.session.execute(sql_receivers)
for row in results:
row = [f"'{r}'" if r else "DEFAULT" for r in row]
receiver_values.append(f"({','.join(row)})")
# Third: get the airports
sql_airports = f"""
SELECT DISTINCT a.name, a.location, a.altitude, a.style, a.border
FROM airports AS a, receivers AS r
WHERE
r.name IN ({','.join(receiver_names)})
AND ST_Within(r.location, ST_Buffer(a.location, 0.2))
AND a.style IN (2,4,5);
"""
airport_values = []
results = db.session.execute(sql_airports)
for row in results:
row = [f"'{r}'" if r else "DEFAULT" for r in row]
airport_values.append(f"({','.join(row)})")
# Last: write all into file
with open(f'{start}_{end}_{name}.sql', 'w') as file:
file.write('/*\n')
file.write('OGN Python SQL Export\n')
file.write(f'Created by: {os.getlogin()}\n')
file.write(f'Created at: {datetime.datetime.utcnow()}\n')
file.write('*/\n\n')
file.write("INSERT INTO airports(name, location, altitude, style, border) VALUES\n")
file.write(',\n'.join(airport_values) + ';\n\n')
file.write("INSERT INTO receivers(name, location) VALUES\n")
file.write(',\n'.join(receiver_values) + ';\n\n')
file.write("INSERT INTO sender_positions(reference_timestamp, name, receiver_name, timestamp, location, track, ground_speed, altitude, aircraft_type, climb_rate, turn_rate, distance, bearing, agl) VALUES\n")
file.write(',\n'.join(sender_position_values) + ';\n\n')
@user_cli.command("cup")
def cup():
"""Export receiver waypoints as '.cup'."""
@ -60,18 +134,18 @@ def cup():
@click.argument("date")
def igc(address, date):
"""Export igc file for <address> at <date>."""
if not re.match(".{6}", address):
print("Address {} not valid.".format(address))
if not re.match("[0-9A-F]{6}", address):
print(f"Address '{address}' not valid.")
return
try:
sender = db.session.query(Sender).filter(Sender.address == address).one()
except NoResultFound as e:
print(f"No data for '{address}' in the DB")
return
if not re.match(r"\d{4}-\d{2}-\d{2}", date):
print("Date {} not valid.".format(date))
return
device_id = db.session.query(Device.id).filter(Device.address == address).first()
if device_id is None:
print("Device with address '{}' not found.".format(address))
print(f"Date {date} not valid.")
return
with open("sample.igc", "wb") as fp:
@ -83,27 +157,26 @@ def igc(address, date):
"logger_id": "OGN",
"date": datetime.date(1987, 2, 24),
"fix_accuracy": 50,
"pilot": "Konstantin Gruendger",
"pilot": "Unknown",
"copilot": "",
"glider_type": "Duo Discus",
"glider_id": "D-KKHH",
"firmware_version": "2.2",
"hardware_version": "2",
"logger_type": "LXNAVIGATION,LX8000F",
"gps_receiver": "uBLOX LEA-4S-2,16,max9000m",
"pressure_sensor": "INTERSEMA,MS5534A,max10000m",
"competition_id": "2H",
"competition_class": "Doubleseater",
"glider_type": sender.infos[0].aircraft if len(sender.infos) > 0 else '',
"glider_id": sender.infos[0].registration if len(sender.infos) > 0 else '',
"firmware_version": sender.software_version,
"hardware_version": sender.hardware_version,
"logger_type": "OGN",
"gps_receiver": "unknown",
"pressure_sensor": "unknown",
"competition_id": sender.infos[0].competition if len(sender.infos) > 0 else '',
"competition_class": "unknown",
}
)
points = (
db.session.query(AircraftBeacon)
.filter(AircraftBeacon.device_id == device_id)
.filter(AircraftBeacon.timestamp > date + " 00:00:00")
.filter(AircraftBeacon.timestamp < date + " 23:59:59")
.order_by(AircraftBeacon.timestamp)
db.session.query(SenderPosition)
.filter(db.between(SenderPosition.reference_timestamp, f"{date} 00:00:00", f"{date} 23:59:59"))
.filter(SenderPosition.name == sender.name)
.order_by(SenderPosition.timestamp)
)
for point in points.all():
for point in points:
writer.write_fix(point.timestamp.time(), latitude=point.location.latitude, longitude=point.location.longitude, valid=True, pressure_alt=point.altitude, gps_alt=point.altitude)

114
app/commands/flights.py
Wyświetl plik

@ -6,119 +6,11 @@ from tqdm import tqdm
from app.commands.database import get_database_days
from app import db
from app.collect.flights import compute_flights, compute_gaps
user_cli = AppGroup("flights")
user_cli.help = "Create 2D flight paths from data."
NOTHING = ""
CONTEST_RELEVANT = "AND agl < 1000"
LOW_PASS = "AND agl < 50 and ground_speed > 250"
def compute_gaps(session, date):
query = """
INSERT INTO flights2d(date, flight_type, device_id, path)
SELECT '{date}' AS date,
3 AS flight_type,
sq3.device_id,
ST_Collect(sq3.path)
FROM (
SELECT sq2.d1 device_id,
ST_MakeLine(sq2.l1, sq2.l2) path
FROM
(
SELECT sq.timestamp t1,
LAG(sq.timestamp) OVER ( PARTITION BY sq.timestamp::DATE, sq.device_id ORDER BY sq.timestamp) t2,
sq.location l1,
LAG(sq.location) OVER ( PARTITION BY sq.timestamp::DATE, sq.device_id ORDER BY sq.timestamp) l2,
sq.device_id d1,
LAG(sq.device_id) OVER ( PARTITION BY sq.timestamp::DATE, sq.device_id ORDER BY sq.timestamp) d2
FROM
(
SELECT DISTINCT ON (device_id, timestamp) timestamp, device_id, location, agl
FROM aircraft_beacons
WHERE timestamp BETWEEN '{date} 00:00:00' AND '{date} 23:59:59' AND agl > 300
ORDER BY device_id, timestamp, error_count
) sq
) sq2
WHERE EXTRACT(epoch FROM sq2.t1 - sq2.t2) > 300
AND ST_DistanceSphere(sq2.l1, sq2.l2) / EXTRACT(epoch FROM sq2.t1 - sq2.t2) BETWEEN 15 AND 50
) sq3
GROUP BY sq3.device_id
ON CONFLICT DO NOTHING;
""".format(
date=date.strftime("%Y-%m-%d")
)
session.execute(query)
session.commit()
def compute_flights2d(session, date, flight_type):
if flight_type == 0:
filter = NOTHING
elif flight_type == 1:
filter = CONTEST_RELEVANT
elif flight_type == 2:
filter = LOW_PASS
query = """
INSERT INTO flights2d
(
date,
flight_type,
device_id,
path,
path_simple
)
SELECT '{date}' AS date,
{flight_type} as flight_type,
sq5.device_id,
st_collect(sq5.linestring order BY sq5.part) multilinestring,
st_collect(st_simplify(sq5.linestring, 0.0001) ORDER BY sq5.part) simple_multilinestring
FROM (
SELECT sq4.device_id,
sq4.part,
st_makeline(sq4.location ORDER BY sq4.timestamp) linestring
FROM (
SELECT sq3.timestamp,
sq3.location,
sq3.device_id,
sum(sq3.ping) OVER (partition BY sq3.device_id ORDER BY sq3.timestamp) part
FROM (
SELECT sq2.t1 AS timestamp,
sq2.l1 AS location,
sq2.d1 device_id,
CASE
WHEN sq2.t1 - sq2.t2 < interval'100s' AND ST_DistanceSphere(sq2.l1, sq2.l2) < 1000 THEN 0
ELSE 1
END AS ping
FROM (
SELECT sq.timestamp t1,
lag(sq.timestamp) OVER (partition BY sq.device_id ORDER BY sq.timestamp) t2,
sq.location l1,
lag(sq.location) OVER (partition BY sq.device_id ORDER BY sq.timestamp) l2,
sq.device_id d1,
lag(sq.device_id) OVER (partition BY sq.device_id ORDER BY sq.timestamp) d2
FROM (
SELECT DISTINCT ON (device_id, timestamp) timestamp, device_id, location
FROM aircraft_beacons
WHERE timestamp BETWEEN '{date} 00:00:00' AND '{date} 23:59:59' {filter}
ORDER BY device_id, timestamp, error_count
) sq
) sq2
) sq3
) sq4
GROUP BY sq4.device_id, sq4.part
) sq5
GROUP BY sq5.device_id
ON CONFLICT DO NOTHING;
""".format(
date=date.strftime("%Y-%m-%d"), flight_type=flight_type, filter=filter
)
session.execute(query)
session.commit()
@user_cli.command("create")
@click.argument("start")
@ -133,6 +25,6 @@ def create(start, end, flight_type):
for single_date in pbar:
pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d"))
if flight_type <= 2:
result = compute_flights2d(session=db.session, date=single_date, flight_type=flight_type)
result = compute_flights(date=single_date, flight_type=flight_type)
else:
result = compute_gaps(session=db.session, date=single_date)
result = compute_gaps(date=single_date)

102
app/commands/gateway.py
Wyświetl plik

@ -1,49 +1,103 @@
import os
import datetime
from datetime import datetime, timezone
import time
from flask import current_app
from flask.cli import AppGroup
import click
from tqdm import tqdm
from ogn.client import AprsClient
from app.gateway.bulkimport import convert, DbFeeder
from app import redis_client
from app.gateway.beacon_conversion import aprs_string_to_message
from app.gateway.message_handling import receiver_status_message_to_csv_string, receiver_position_message_to_csv_string, sender_position_message_to_csv_string
from app.collect.gateway import transfer_from_redis_to_database
user_cli = AppGroup("gateway")
user_cli.help = "Connection to APRS servers."
@user_cli.command("run")
def run(aprs_user="anon-dev"):
"""Run the aprs client and feed the DB with incoming data."""
@click.option("--aprs_filter", default='')
def run(aprs_filter):
"""
Run the aprs client, parse the incoming data and put it to redis.
"""
# User input validation
if len(aprs_user) < 3 or len(aprs_user) > 9:
print("aprs_user must be a string of 3-9 characters.")
return
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(name)-17s %(levelname)-8s %(message)s')
current_app.logger.warning("Start ogn gateway")
client = AprsClient(aprs_user)
client = AprsClient(current_app.config['APRS_USER'], aprs_filter)
client.connect()
with DbFeeder(prefix='continuous_import', reference_timestamp=datetime.utcnow, reference_timestamp_autoupdate=True) as feeder:
try:
client.run(callback=lambda x: feeder.add(x), autoreconnect=True)
except KeyboardInterrupt:
current_app.logger.warning("\nStop ogn gateway")
def insert_into_redis(aprs_string):
# Convert aprs_string to message dict, add MGRS Position, flatten gps precision, etc. etc. ...
message = aprs_string_to_message(aprs_string)
if message is None:
return
# separate between tables (receiver/sender) and aprs_type (status/position)
if message['beacon_type'] in ('aprs_receiver', 'receiver'):
if message['aprs_type'] == 'status':
redis_target = 'receiver_status'
csv_string = receiver_status_message_to_csv_string(message, none_character=r'\N')
elif message['aprs_type'] == 'position':
redis_target = 'receiver_position'
csv_string = receiver_position_message_to_csv_string(message, none_character=r'\N')
else:
return
else:
if message['aprs_type'] == 'status':
return # no interesting data we want to keep
elif message['aprs_type'] == 'position':
redis_target = 'sender_position'
csv_string = sender_position_message_to_csv_string(message, none_character=r'\N')
else:
return
mapping = {csv_string: str(time.time())}
redis_client.zadd(name=redis_target, mapping=mapping, nx=True)
insert_into_redis.beacon_counter += 1
current_minute = datetime.utcnow().minute
if current_minute != insert_into_redis.last_minute:
current_app.logger.info(f"{insert_into_redis.beacon_counter:7d}/min")
insert_into_redis.beacon_counter = 0
insert_into_redis.last_minute = current_minute
insert_into_redis.beacon_counter = 0
insert_into_redis.last_minute = datetime.utcnow().minute
try:
client.run(callback=insert_into_redis, autoreconnect=True)
except KeyboardInterrupt:
current_app.logger.warning("\nStop ogn gateway")
client.disconnect()
@user_cli.command("convert")
@click.argument("path")
def file_import(path):
"""Convert APRS logfiles into csv files for fast bulk import."""
@user_cli.command("transfer")
def transfer():
"""Transfer data from redis to the database."""
logfiles = []
for (root, dirs, files) in os.walk(path):
for file in sorted(files):
logfiles.append(os.path.join(root, file))
transfer_from_redis_to_database()
for logfile in logfiles:
convert(logfile)
@user_cli.command("printout")
@click.option("--aprs_filter", default='')
def printout(aprs_filter):
"""Run the aprs client and just print out the data stream."""
current_app.logger.warning("Start ogn gateway")
client = AprsClient(current_app.config['APRS_USER'], aprs_filter=aprs_filter)
client.connect()
try:
client.run(callback=lambda x: print(f"{datetime.utcnow()}: {x}"), autoreconnect=True)
except KeyboardInterrupt:
current_app.logger.warning("\nStop ogn gateway")
client.disconnect()

86
app/commands/logbook.py
Wyświetl plik

@ -3,18 +3,13 @@ import click
from datetime import datetime
from app.collect.logbook import update_entries as logbook_update_entries
from app.collect.takeoff_landings import update_entries as takeoff_landings_update_entries
from app.model import Airport, Logbook
from sqlalchemy.sql import func
from app.collect.logbook import update_takeoff_landings, update_logbook
from tqdm import tqdm
from app.commands.database import get_database_days
from app.utils import date_to_timestamps
from app import db
user_cli = AppGroup("logbook")
user_cli.help = "Handling of logbook data."
user_cli.help = "Handling of takeoff/landings and logbook data."
@user_cli.command("compute_takeoff_landing")
@ -29,7 +24,7 @@ def compute_takeoff_landing(start, end):
for single_date in pbar:
pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d"))
(start, end) = date_to_timestamps(single_date)
result = takeoff_landings_update_entries(session=db.session, start=start, end=end)
result = update_takeoff_landings(start=start, end=end)
@user_cli.command("compute_logbook")
@ -43,77 +38,4 @@ def compute_logbook(start, end):
pbar = tqdm(days)
for single_date in pbar:
pbar.set_description(single_date.strftime("%Y-%m-%d"))
result = logbook_update_entries(session=db.session, date=single_date)
@user_cli.command("show")
@click.argument("airport_name")
@click.argument("date")
def show(airport_name, date=None):
"""Show a logbook for <airport_name>."""
airport = db.session.query(Airport).filter(Airport.name == airport_name).first()
if airport is None:
print('Airport "{}" not found.'.format(airport_name))
return
or_args = []
if date is not None:
date = datetime.strptime(date, "%Y-%m-%d")
(start, end) = date_to_timestamps(date)
or_args = [db.between(Logbook.reftime, start, end)]
# get all logbook entries and add device and airport infos
logbook_query = (
db.session.query(func.row_number().over(order_by=Logbook.reftime).label("row_number"), Logbook)
.filter(*or_args)
.filter(db.or_(Logbook.takeoff_airport_id == airport.id, Logbook.landing_airport_id == airport.id))
.order_by(Logbook.reftime)
)
# ... and finally print out the logbook
print("--- Logbook ({}) ---".format(airport_name))
def none_datetime_replacer(datetime_object):
return "--:--:--" if datetime_object is None else datetime_object.time()
def none_track_replacer(track_object):
return "--" if track_object is None else round(track_object / 10.0)
def none_timedelta_replacer(timedelta_object):
return "--:--:--" if timedelta_object is None else timedelta_object
def none_registration_replacer(device_object):
return "[" + device_object.address + "]" if len(device_object.infos) == 0 else device_object.infos[0].registration
def none_aircraft_replacer(device_object):
return "(unknown)" if len(device_object.infos) == 0 else device_object.infos[0].aircraft
def airport_marker(logbook_object):
if logbook_object.takeoff_airport is not None and logbook_object.takeoff_airport.name is not airport.name:
return "FROM: {}".format(logbook_object.takeoff_airport.name)
elif logbook_object.landing_airport is not None and logbook_object.landing_airport.name is not airport.name:
return "TO: {}".format(logbook_object.landing_airport.name)
else:
return ""
def none_altitude_replacer(logbook_object):
return "?" if logbook_object.max_altitude is None else "{:5d}m ({:+5d}m)".format(logbook_object.max_altitude, logbook_object.max_altitude - logbook_object.takeoff_airport.altitude)
for [row_number, logbook] in logbook_query.all():
print(
"%3d. %10s %8s (%2s) %8s (%2s) %8s %15s %8s %17s %20s"
% (
row_number,
logbook.reftime.date(),
none_datetime_replacer(logbook.takeoff_timestamp),
none_track_replacer(logbook.takeoff_track),
none_datetime_replacer(logbook.landing_timestamp),
none_track_replacer(logbook.landing_track),
none_timedelta_replacer(logbook.duration),
none_altitude_replacer(logbook),
none_registration_replacer(logbook.device),
none_aircraft_replacer(logbook.device),
airport_marker(logbook),
)
)
result = update_logbook(date=single_date)

130
app/commands/stats.py
Wyświetl plik

@ -1,130 +0,0 @@
from flask.cli import AppGroup
import click
from datetime import datetime
from tqdm import tqdm
from app.commands.database import get_database_days
from app.collect.stats import (
create_device_stats,
create_receiver_stats,
create_relation_stats,
create_country_stats,
update_qualities,
update_receivers as update_receivers_command,
update_devices as update_devices_command,
update_device_stats_jumps,
)
from app.collect.ognrange import update_entries as update_receiver_coverages
from app.model import Device
from app import db
user_cli = AppGroup("stats")
user_cli.help = "Handling of statistical data."
@user_cli.command("create")
@click.argument("start")
@click.argument("end")
def create(start, end):
"""Create DeviceStats, ReceiverStats and RelationStats."""
days = get_database_days(start, end)
pbar = tqdm(days)
for single_date in pbar:
pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d"))
result = create_device_stats(session=db.session, date=single_date)
result = update_device_stats_jumps(session=db.session, date=single_date)
result = create_receiver_stats(session=db.session, date=single_date)
result = create_relation_stats(session=db.session, date=single_date)
result = update_qualities(session=db.session, date=single_date)
@user_cli.command("create_country")
@click.argument("start")
@click.argument("end")
def create_country(start, end):
"""Create CountryStats."""
days = get_database_days(start, end)
pbar = tqdm(days)
for single_date in pbar:
pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d"))
result = create_country_stats(session=db.session, date=single_date)
@user_cli.command("update_devices_name")
def update_devices_name():
"""Update Devices name."""
device_ids = db.session.query(Device.id).all()
for device_id in tqdm(device_ids):
db.session.execute(
"update devices d set name = sq.name from ( select * from aircraft_beacons ab where ab.device_id = {} limit 1) sq where d.id = sq.device_id and d.name is null or d.name = 'ICA3D3CC4';".format(
device_id[0]
)
)
db.session.commit()
@user_cli.command("update_receivers")
def update_receivers():
"""Update receivers with data from stats."""
result = update_receivers_command(session=db.session)
print(result)
@user_cli.command("update_devices")
def update_devices():
"""Update devices with data from stats."""
result = update_devices_command(session=db.session)
print(result)
@user_cli.command("update_mgrs")
@click.argument("start")
@click.argument("end")
def update_mgrs(start, end):
"""Create location_mgrs_short."""
days = get_database_days(start, end)
pbar = tqdm(days)
for single_date in pbar:
datestr = datetime.strftime(single_date, "%Y-%m-%d")
pbar.set_description(datestr)
for pbar2 in tqdm(["{:02d}:{:02d}".format(hh, mm) for hh in range(0, 24) for mm in range(0, 60)]):
sql = """
UPDATE aircraft_beacons
SET location_mgrs_short = left(location_mgrs, 5) || substring(location_mgrs, 6, 2) || substring(location_mgrs, 11, 2)
WHERE timestamp BETWEEN '{0} {1}:00' and '{0} {1}:59' AND location_mgrs_short IS NULL;
""".format(
datestr, pbar2
)
# print(sql)
db.session.execute(sql)
db.session.commit()
@user_cli.command("create_ognrange")
@click.argument("start")
@click.argument("end")
def create_ognrange(start=None, end=None):
"""Create receiver coverage stats for Melissas ognrange."""
days = get_database_days(start, end)
pbar = tqdm(days)
for single_date in pbar:
pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d"))
result = update_receiver_coverages(session=db.session, date=single_date)

0
app/config/__init__.py
Wyświetl plik

28
app/config/default.py
Wyświetl plik

@ -1,28 +0,0 @@
import os
SECRET_KEY = "i-like-ogn"
SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI", "postgresql://postgres@localhost:5432/ogn")
SQLALCHEMY_TRACK_MODIFICATIONS = False
# Flask-Cache stuff
CACHE_TYPE = "simple"
CACHE_DEFAULT_TIMEOUT = 300
# Celery stuff
CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND", "redis://localhost:6379/0")
from celery.schedules import crontab
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
"update-ddb": {"task": "import_ddb", "schedule": timedelta(hours=1)},
"update-country-codes": {"task": "update_receivers_country_code", "schedule": timedelta(days=1)},
"update-takeoff-and-landing": {"task": "update_takeoff_landings", "schedule": timedelta(hours=1), "kwargs": {"last_minutes": 90}},
"update-logbook": {"task": "update_logbook_entries", "schedule": timedelta(hours=2), "kwargs": {"day_offset": 0}},
"update-max-altitudes": {"task": "update_logbook_max_altitude", "schedule": timedelta(hours=1), "kwargs": {"day_offset": 0}},
"update-stats-daily": {"task": "update_stats", "schedule": crontab(hour=0, minute=5), "kwargs": {"day_offset": -1}},
"update-logbook-daily": {"task": "update_logbook_entries", "schedule": crontab(hour=1, minute=0), "kwargs": {"day_offset": -1}},
"purge_old_data": {"task": "purge_old_data", "schedule": timedelta(hours=1), "kwargs": {"max_hours": 48}},
}

7
app/config/test.py
Wyświetl plik

@ -1,7 +0,0 @@
SQLALCHEMY_DATABASE_URI = "postgresql://postgres@localhost:5432/ogn_test"
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_ECHO = True
# Celery stuff
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/0"

52
app/gateway/beacon_conversion.py 100644
Wyświetl plik

@ -0,0 +1,52 @@
from flask import current_app
from mgrs import MGRS
from ogn.parser import parse
from app.model import AircraftType
#import rasterio as rs
#elevation_dataset = rs.open('/Volumes/LaCieBlack/Wtf4.tiff')
mgrs = MGRS()
def aprs_string_to_message(aprs_string):
try:
message = parse(aprs_string, calculate_relations=True)
except Exception as e:
current_app.logger.debug(e)
return None
if message['aprs_type'] not in ('position', 'status'):
return None
elif message['aprs_type'] == 'position':
latitude = message["latitude"]
longitude = message["longitude"]
message["location"] = "SRID=4326;POINT({} {})".format(longitude, latitude)
location_mgrs = mgrs.toMGRS(latitude, longitude).decode("utf-8")
message["location_mgrs"] = location_mgrs
message["location_mgrs_short"] = location_mgrs[0:5] + location_mgrs[5:7] + location_mgrs[10:12]
#if 'altitude' in message and longitude >= 0.0 and longitude <= 20.0 and latitude >= 40.0 and latitude <= 60.0:
# elevation = [val[0] for val in elevation_dataset.sample(((longitude, latitude),))][0]
# message['agl'] = message['altitude'] - elevation
if 'bearing' in message:
bearing = int(message['bearing'])
message['bearing'] = bearing if bearing < 360 else 0
if "aircraft_type" in message:
message["aircraft_type"] = AircraftType(message["aircraft_type"]) if message["aircraft_type"] in AircraftType.list() else AircraftType.UNKNOWN
if "gps_quality" in message:
if message["gps_quality"] is not None and "horizontal" in message["gps_quality"]:
message["gps_quality_horizontal"] = message["gps_quality"]["horizontal"]
message["gps_quality_vertical"] = message["gps_quality"]["vertical"]
del message["gps_quality"]
return message

206
app/gateway/bulkimport.py
Wyświetl plik

@ -1,206 +0,0 @@
import os
import re
from datetime import datetime, timedelta
from io import StringIO
from flask import current_app
from flask.cli import AppGroup
import click
from tqdm import tqdm
from mgrs import MGRS
from ogn.parser import parse, ParseError
from app.model import AircraftType, Location
from app.gateway.process_tools import open_file, create_tables, drop_tables, update_aircraft_beacons_bigdata
from app import db
user_cli = AppGroup("bulkimport")
user_cli.help = "Tools for accelerated data import."
basepath = os.path.dirname(os.path.realpath(__file__))
# define message types we want to proceed
AIRCRAFT_BEACON_TYPES = ["aprs_aircraft", "flarm", "tracker", "fanet", "lt24", "naviter", "skylines", "spider", "spot", "flymaster"]
RECEIVER_BEACON_TYPES = ["aprs_receiver", "receiver"]
# define fields we want to proceed
BEACON_KEY_FIELDS = ["name", "receiver_name", "timestamp"]
AIRCRAFT_BEACON_FIELDS = [
"location",
"altitude",
"dstcall",
"relay",
"track",
"ground_speed",
"address_type",
"aircraft_type",
"stealth",
"address",
"climb_rate",
"turn_rate",
"signal_quality",
"error_count",
"frequency_offset",
"gps_quality_horizontal",
"gps_quality_vertical",
"software_version",
"hardware_version",
"real_address",
"signal_power",
"distance",
"radial",
"quality",
"location_mgrs",
"location_mgrs_short",
"agl",
]
RECEIVER_BEACON_FIELDS = [
"location",
"altitude",
"dstcall",
]
def initial_file_scan(file):
"""Scan file and get rowcount and first server timestamp."""
row_count = 0
timestamp = None
for row in file:
row_count += 1
if timestamp is None and row[0] == '#':
message = parse(row)
if message['aprs_type'] == 'server':
timestamp = message['timestamp']
file.seek(0)
return row_count, timestamp
class DbFeeder:
def __init__(self, postfix, reference_timestamp, auto_update_timestamp):
self.postfix = postfix
self.reference_timestamp = reference_timestamp
self.auto_update_timestamp = auto_update_timestamp
self.last_flush = datetime.utcnow()
self.aircraft_buffer = StringIO()
self.receiver_buffer = StringIO()
self.connection = db.engine.raw_connection()
self.cursor = self.connection.cursor()
self.mgrs = MGRS()
create_tables(self.postfix)
def __enter__(self):
return self
def __exit__(self, *args):
self._flush()
update_aircraft_beacons_bigdata(self.postfix)
self.connection.commit()
self.cursor.close()
self.connection.close()
def _flush(self):
self.aircraft_buffer.seek(0)
self.receiver_buffer.seek(0)
self.cursor.copy_from(self.aircraft_buffer, "aircraft_beacons_{postfix}".format(postfix=self.postfix), sep=",", columns=BEACON_KEY_FIELDS + AIRCRAFT_BEACON_FIELDS)
self.cursor.copy_from(self.receiver_buffer, "receiver_beacons_{postfix}".format(postfix=self.postfix), sep=",", columns=BEACON_KEY_FIELDS + RECEIVER_BEACON_FIELDS)
self.connection.commit()
self.aircraft_buffer = StringIO()
self.receiver_buffer = StringIO()
def add(self, raw_string):
try:
message = parse(raw_string, reference_timestamp=self.reference_timestamp)
except NotImplementedError as e:
current_app.logger.error("No parser implemented for message: {}".format(raw_string))
return
except ParseError as e:
current_app.logger.error("Parsing error with message: {}".format(raw_string))
return
except TypeError as e:
current_app.logger.error("TypeError with message: {}".format(raw_string))
return
except Exception as e:
current_app.logger.error("Other Exception with string: {}".format(raw_string))
return
if message['aprs_type'] not in ('server', 'position'):
return
elif message['aprs_type'] == 'server' and self.auto_update_timestamp is True:
self.reference_timestamp = message['timestamp']
return
elif message['aprs_type'] == 'position':
latitude = message["latitude"]
longitude = message["longitude"]
location = Location(longitude, latitude)
message["location"] = location.to_wkt()
location_mgrs = self.mgrs.toMGRS(latitude, longitude).decode("utf-8")
message["location_mgrs"] = location_mgrs
message["location_mgrs_short"] = location_mgrs[0:5] + location_mgrs[5:7] + location_mgrs[10:12]
if "aircraft_type" in message:
message["aircraft_type"] = AircraftType(message["aircraft_type"]).name if message["aircraft_type"] in AircraftType.list() else AircraftType.UNKNOWN.name
if "gps_quality" in message:
if message["gps_quality"] is not None and "horizontal" in message["gps_quality"]:
message["gps_quality_horizontal"] = message["gps_quality"]["horizontal"]
message["gps_quality_vertical"] = message["gps_quality"]["vertical"]
del message["gps_quality"]
if message["beacon_type"] in RECEIVER_BEACON_TYPES:
complete_message = ",".join([str(message[k]) if k in message and message[k] is not None else "\\N" for k in BEACON_KEY_FIELDS + RECEIVER_BEACON_FIELDS])
self.receiver_buffer.write(complete_message)
self.receiver_buffer.write("\n")
elif message["beacon_type"] in AIRCRAFT_BEACON_TYPES:
complete_message = ",".join([str(message[k]) if k in message and message[k] is not None else "\\N" for k in BEACON_KEY_FIELDS + AIRCRAFT_BEACON_FIELDS])
self.aircraft_buffer.write(complete_message)
self.aircraft_buffer.write("\n")
else:
current_app.logger.error("Ignore beacon_type: {}".format(message["beacon_type"]))
return
if datetime.utcnow() - self.last_flush >= timedelta(seconds=5):
self._flush()
self.last_flush = datetime.utcnow()
def convert(sourcefile):
with open_file(sourcefile) as filehandler:
total_lines, reference_timestamp = initial_file_scan(filehandler)
if reference_timestamp is not None:
auto_update_timestamp = True
postfix = str(reference_timestamp.total_seconds())
else:
auto_update_timestamp = False
match = re.match(r".*OGN_log\.txt_([0-9]{4}\-[0-9]{2}\-[0-9]{2})\.gz$", sourcefile)
if match:
reference_timestamp = datetime.strptime(match.group(1), "%Y-%m-%d")
postfix = reference_timestamp.strftime("%Y_%m_%d")
else:
current_app.logger.error("No reference time information. Skipping file: {}".format(sourcefile))
return
with open_file(sourcefile) as fin:
with DbFeeder(postfix=postfix, reference_timestamp=reference_timestamp, auto_update_timestamp=auto_update_timestamp) as feeder:
pbar = tqdm(fin, total=total_lines)
for line in pbar:
pbar.set_description("Importing {}".format(sourcefile))
feeder.add(raw_string=line)

438
app/gateway/message_handling.py 100644
Wyświetl plik

@ -0,0 +1,438 @@
import os
import time
from io import StringIO
from app import db
from app.model import AircraftType
from app.utils import get_sql_trustworthy
basepath = os.path.dirname(os.path.realpath(__file__))
# define fields we want to proceed
SENDER_POSITION_BEACON_FIELDS = [
"reference_timestamp",
"name",
"dstcall",
"relay",
"receiver_name",
"timestamp",
"location",
"track",
"ground_speed",
"altitude",
"address_type",
"aircraft_type",
"stealth",
"address",
"climb_rate",
"turn_rate",
"signal_quality",
"error_count",
"frequency_offset",
"gps_quality_horizontal",
"gps_quality_vertical",
"software_version",
"hardware_version",
"real_address",
"signal_power",
"distance",
"bearing",
"normalized_quality",
"location_mgrs",
"location_mgrs_short",
"agl",
]
RECEIVER_POSITION_BEACON_FIELDS = [
"reference_timestamp",
"name",
"dstcall",
"receiver_name",
"timestamp",
"location",
"altitude",
"location_mgrs",
"location_mgrs_short",
"agl",
]
RECEIVER_STATUS_BEACON_FIELDS = [
"reference_timestamp",
"name",
"dstcall",
"receiver_name",
"timestamp",
"version",
"platform",
"cpu_temp",
"rec_input_noise",
]
def sender_position_message_to_csv_string(message, none_character=''):
"""
Convert sender_position_messages to csv string.
:param dict message: dict of sender position messages from the parser
:param str none_character: '' for a file, '\\N' for Postgresql COPY
"""
csv_string = "{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11},{12},{13},{14},{15},{16},{17},{18},{19},{20},{21},{22},{23},{24},{25},{26},{27},{28},{29},{30}\n".format(
message['reference_timestamp'],
message['name'],
message['dstcall'],
message['relay'] if 'relay' in message and message['relay'] else none_character,
message['receiver_name'],
message['timestamp'],
message['location'],
message['track'] if 'track' in message and message['track'] else none_character,
message['ground_speed'] if 'ground_speed' in message and message['ground_speed'] else none_character,
int(message['altitude']) if message['altitude'] else none_character,
message['address_type'] if 'address_type' in message and message['address_type'] else none_character, # 10
message['aircraft_type'].name if 'aircraft_type' in message and message['aircraft_type'] else AircraftType.UNKNOWN.name,
message['stealth'] if 'stealth' in message and message['stealth'] else none_character,
message['address'] if 'address' in message and message['address'] else none_character,
message['climb_rate'] if 'climb_rate' in message and message['climb_rate'] else none_character,
message['turn_rate'] if 'turn_rate' in message and message['turn_rate'] else none_character,
message['signal_quality'] if 'signal_quality' in message and message['signal_quality'] else none_character,
message['error_count'] if 'error_count' in message and message['error_count'] else none_character,
message['frequency_offset'] if 'frequency_offset' in message and message['frequency_offset'] else none_character,
message['gps_quality_horizontal'] if 'gps_quality_horizontal' in message and message['gps_quality_horizontal'] else none_character,
message['gps_quality_vertical'] if 'gps_quality_vertical' in message and message['gps_quality_vertical'] else none_character, # 20
message['software_version'] if 'software_version' in message and message['software_version'] else none_character,
message['hardware_version'] if 'hardware_version' in message and message['hardware_version'] else none_character,
message['real_address'] if 'real_address' in message and message['real_address'] else none_character,
message['signal_power'] if 'signal_power' in message and message['signal_power'] else none_character,
message['distance'] if 'distance' in message and message['distance'] else none_character,
message['bearing'] if 'bearing' in message and message['bearing'] else none_character,
message['normalized_quality'] if 'normalized_quality' in message and message['normalized_quality'] else none_character,
message['location_mgrs'],
message['location_mgrs_short'],
message['agl'] if 'agl' in message else none_character,
)
return csv_string
def receiver_position_message_to_csv_string(message, none_character=''):
csv_string = "{0},{1},{2},{3},{4},{5},{6},{7},{8},{9}\n".format(
message['reference_timestamp'],
message['name'],
message['dstcall'],
message['receiver_name'],
message['timestamp'],
message['location'],
int(message['altitude']) if message['altitude'] else none_character,
message['location_mgrs'],
message['location_mgrs_short'],
message['agl'] if 'agl' in message else none_character,
)
return csv_string
def receiver_status_message_to_csv_string(message, none_character=''):
csv_string = "{0},{1},{2},{3},{4},{5},{6},{7},{8}\n".format(
message['reference_timestamp'],
message['name'],
message['dstcall'],
message['receiver_name'],
message['timestamp'],
message['version'] if 'version' in message else none_character,
message['platform'] if 'platform' in message else none_character,
message['cpu_temp'] if 'cpu_temp' in message else none_character,
message['rec_input_noise'] if 'rec_input_noise' in message else none_character,
)
return csv_string
def sender_position_csv_strings_to_db(lines):
timestamp_string = str(time.time()).replace('.', '_')
tmp_tablename = f'sender_positions_{timestamp_string}'
connection = db.engine.raw_connection()
cursor = connection.cursor()
string_buffer = StringIO()
string_buffer.writelines(lines)
string_buffer.seek(0)
cursor.execute(f"CREATE TEMPORARY TABLE {tmp_tablename} (LIKE sender_positions) ON COMMIT DROP;")
cursor.copy_from(file=string_buffer, table=tmp_tablename, sep=",", columns=SENDER_POSITION_BEACON_FIELDS)
# Update agl
cursor.execute(f"""
UPDATE {tmp_tablename} AS tmp
SET
agl = tmp.altitude - ST_Value(e.rast, tmp.location)
FROM elevation AS e
WHERE ST_Intersects(tmp.location, e.rast);
""")
# Update sender position statistics
cursor.execute(f"""
INSERT INTO sender_position_statistics AS sps (date, dstcall, address_type, aircraft_type, stealth, software_version, hardware_version, messages_count)
SELECT
tmp.reference_timestamp::DATE AS date,
tmp.dstcall,
tmp.address_type,
tmp.aircraft_type,
tmp.stealth,
tmp.software_version,
tmp.hardware_version,
COUNT(tmp.*) AS messages_count
FROM {tmp_tablename} AS tmp
GROUP BY date, dstcall, address_type, aircraft_type, stealth, software_version, hardware_version
ON CONFLICT (date, dstcall, address_type, aircraft_type, stealth, software_version, hardware_version) DO UPDATE
SET
messages_count = EXCLUDED.messages_count + sps.messages_count;
""")
# Update senders
cursor.execute(f"""
INSERT INTO senders AS s (firstseen, lastseen, name, aircraft_type, stealth, address, software_version, hardware_version, real_address)
SELECT DISTINCT ON (tmp.name)
tmp.reference_timestamp AS firstseen,
tmp.reference_timestamp AS lastseen,
tmp.name,
tmp.aircraft_type,
tmp.stealth,
tmp.address,
tmp.software_version,
tmp.hardware_version,
tmp.real_address
FROM {tmp_tablename} AS tmp
WHERE tmp.name NOT LIKE 'RND%'
ON CONFLICT (name) DO UPDATE
SET
lastseen = GREATEST(EXCLUDED.lastseen, s.lastseen),
aircraft_type = EXCLUDED.aircraft_type,
stealth = EXCLUDED.stealth,
address = EXCLUDED.address,
software_version = COALESCE(EXCLUDED.software_version, s.software_version),
hardware_version = COALESCE(EXCLUDED.hardware_version, s.hardware_version),
real_address = COALESCE(EXCLUDED.real_address, s.real_address);
""")
# Update sender_infos FK -> senders
cursor.execute("""
UPDATE sender_infos AS si
SET sender_id = s.id
FROM senders AS s
WHERE si.sender_id IS NULL AND s.address = si.address;
""")
SQL_TRUSTWORTHY = get_sql_trustworthy(source_table_alias='tmp')
# Update coverage statistics
cursor.execute(f"""
INSERT INTO coverage_statistics AS rs (date, location_mgrs_short, sender_id, receiver_id, is_trustworthy, max_distance, max_normalized_quality, messages_count)
SELECT
tmp.reference_timestamp::DATE AS date,
tmp.location_mgrs_short,
tmp.sender_id,
tmp.receiver_id,
({SQL_TRUSTWORTHY}) AS is_trustworthy,
MAX(tmp.distance) AS max_distance,
MAX(tmp.normalized_quality) AS max_normalized_quality,
COUNT(tmp.*) AS messages_count
FROM (SELECT x.*, s.id AS sender_id, r.id AS receiver_id FROM {tmp_tablename} AS x INNER JOIN senders AS s ON x.name = s.name INNER JOIN receivers AS r ON x.receiver_name = r.name) AS tmp
GROUP BY date, location_mgrs_short, sender_id, receiver_id, is_trustworthy
ON CONFLICT (date, location_mgrs_short, sender_id, receiver_id, is_trustworthy) DO UPDATE
SET
max_distance = GREATEST(EXCLUDED.max_distance, rs.max_distance),
max_normalized_quality = GREATEST(EXCLUDED.max_normalized_quality, rs.max_normalized_quality),
messages_count = EXCLUDED.messages_count + rs.messages_count;
""")
# Insert all the beacons
all_fields = ', '.join(SENDER_POSITION_BEACON_FIELDS)
cursor.execute(f"""
INSERT INTO sender_positions ({all_fields})
SELECT {all_fields} FROM {tmp_tablename};
""")
connection.commit()
cursor.close()
connection.close()
def receiver_position_csv_strings_to_db(lines):
timestamp_string = str(time.time()).replace('.', '_')
tmp_tablename = f'receiver_positions_{timestamp_string}'
connection = db.engine.raw_connection()
cursor = connection.cursor()
string_buffer = StringIO()
string_buffer.writelines(lines)
string_buffer.seek(0)
cursor.execute(f"CREATE TEMPORARY TABLE {tmp_tablename} (LIKE receiver_positions) ON COMMIT DROP;")
cursor.copy_from(file=string_buffer, table=tmp_tablename, sep=",", columns=RECEIVER_POSITION_BEACON_FIELDS)
# Update agl
cursor.execute(f"""
UPDATE {tmp_tablename} AS tmp
SET
agl = tmp.altitude - ST_Value(e.rast, tmp.location)
FROM elevation AS e
WHERE ST_Intersects(tmp.location, e.rast);
""")
# Update receivers
cursor.execute(f"""
INSERT INTO receivers AS r (firstseen, lastseen, name, timestamp, location, altitude, agl)
SELECT DISTINCT ON (tmp.name)
tmp.reference_timestamp AS firstseen,
tmp.reference_timestamp AS lastseen,
tmp.name,
tmp.timestamp,
tmp.location,
tmp.altitude,
tmp.agl
FROM {tmp_tablename} AS tmp,
(
SELECT
tmp.name,
MAX(timestamp) AS timestamp
FROM {tmp_tablename} AS tmp
GROUP BY tmp.name
) AS sq
WHERE tmp.name = sq.name AND tmp.timestamp = sq.timestamp AND tmp.name NOT LIKE 'RND%'
ON CONFLICT (name) DO UPDATE
SET
lastseen = EXCLUDED.lastseen,
timestamp = EXCLUDED.timestamp,
location = EXCLUDED.location,
altitude = EXCLUDED.altitude,
agl = EXCLUDED.agl;
""")
# Update receiver country
cursor.execute("""
UPDATE receivers AS r
SET
country_id = c.gid
FROM countries AS c
WHERE r.country_id IS NULL AND ST_Within(r.location, c.geom);
""")
# Update receiver airport
cursor.execute("""
UPDATE receivers AS r
SET
airport_id = (
SELECT id
FROM airports AS a
WHERE
ST_Contains(a.border, r.location)
AND a.style IN (2,4,5)
ORDER BY ST_DistanceSphere(a.location, r.location)
LIMIT 1
)
WHERE r.airport_id IS NULL;
""")
# Insert all the beacons
all_fields = ', '.join(RECEIVER_POSITION_BEACON_FIELDS)
cursor.execute(f"""
INSERT INTO receiver_positions ({all_fields})
SELECT {all_fields} FROM {tmp_tablename};
""")
connection.commit()
cursor.close()
connection.close()
def receiver_status_csv_strings_to_db(lines):
timestamp_string = str(time.time()).replace('.', '_')
tmp_tablename = f'receiver_status_{timestamp_string}'
connection = db.engine.raw_connection()
cursor = connection.cursor()
string_buffer = StringIO()
string_buffer.writelines(lines)
string_buffer.seek(0)
cursor.execute(f"CREATE TEMPORARY TABLE {tmp_tablename} (LIKE receiver_statuses) ON COMMIT DROP;")
cursor.copy_from(file=string_buffer, table=tmp_tablename, sep=",", columns=RECEIVER_STATUS_BEACON_FIELDS)
# Update receivers
cursor.execute(f"""
INSERT INTO receivers AS r (firstseen, lastseen, name, timestamp, version, platform, cpu_temp, rec_input_noise)
SELECT DISTINCT ON (tmp.name)
tmp.reference_timestamp AS firstseen,
tmp.reference_timestamp AS lastseen,
tmp.name,
tmp.timestamp,
tmp.version,
tmp.platform,
tmp.cpu_temp,
tmp.rec_input_noise
FROM {tmp_tablename} AS tmp,
(
SELECT
tmp.name,
MAX(timestamp) AS timestamp
FROM {tmp_tablename} AS tmp
GROUP BY tmp.name
) AS sq
WHERE tmp.name = sq.name AND tmp.timestamp = sq.timestamp
ON CONFLICT (name) DO UPDATE
SET
lastseen = EXCLUDED.lastseen,
timestamp = EXCLUDED.timestamp,
version = EXCLUDED.version,
platform = EXCLUDED.platform,
cpu_temp = EXCLUDED.cpu_temp,
rec_input_noise = EXCLUDED.rec_input_noise;
""")
# Insert all the beacons
all_fields = ', '.join(RECEIVER_STATUS_BEACON_FIELDS)
cursor.execute(f"""
INSERT INTO receiver_statuses ({all_fields})
SELECT {all_fields} FROM {tmp_tablename};
""")
connection.commit()
cursor.close()
connection.close()

69
app/gateway/process_tools.py
Wyświetl plik

@ -38,73 +38,10 @@ class Timer(object):
print("Elapsed: {}".format(time.time() - self.tstart))
def drop_tables(postfix):
"""Drop tables for log file import."""
db.session.execute("""
DROP TABLE IF EXISTS "aircraft_beacons_{postfix}";
DROP TABLE IF EXISTS "receiver_beacons_{postfix}";
""".format(postfix=postfix))
db.session.commit()
def create_tables(postfix):
"""Create tables for log file import."""
drop_tables(postfix)
db.session.execute("""
CREATE TABLE aircraft_beacons_{postfix} AS TABLE aircraft_beacons WITH NO DATA;
CREATE TABLE receiver_beacons_{postfix} AS TABLE receiver_beacons WITH NO DATA;
""".format(postfix=postfix))
db.session.commit()
def update_aircraft_beacons_bigdata(postfix):
"""Calculates distance/radial and quality and computes the altitude above ground level.
Due to performance reasons we use a new table instead of updating the old."""
db.session.execute("""
SELECT
ab.location, ab.altitude, ab.name, ab.dstcall, ab.relay, ab.receiver_name, ab.timestamp, ab.track, ab.ground_speed,
ab.address_type, ab.aircraft_type, ab.stealth, ab.address, ab.climb_rate, ab.turn_rate, ab.signal_quality, ab.error_count,
ab.frequency_offset, ab.gps_quality_horizontal, ab.gps_quality_vertical, ab.software_version, ab.hardware_version, ab.real_address, ab.signal_power,
ab.location_mgrs,
ab.location_mgrs_short,
CASE WHEN ab.location IS NOT NULL AND r.location IS NOT NULL THEN CAST(ST_DistanceSphere(ab.location, r.location) AS REAL) ELSE NULL END AS distance,
CASE WHEN ab.location IS NOT NULL AND r.location IS NOT NULL THEN CAST(degrees(ST_Azimuth(ab.location, r.location)) AS SMALLINT) % 360 ELSE NULL END AS radial,
CASE WHEN ab.location IS NOT NULL AND r.location IS NOT NULL AND ST_DistanceSphere(ab.location, r.location) > 0 AND ab.signal_quality IS NOT NULL
THEN CAST(signal_quality + 20*log(ST_DistanceSphere(ab.location, r.location)/10000) AS REAL)
ELSE NULL
END AS quality,
CAST((ab.altitude - subtable.elev_m) AS REAL) AS agl
INTO aircraft_beacons_{postfix}_temp
FROM
aircraft_beacons_{postfix} AS ab
JOIN LATERAL (
SELECT ab.location, MAX(ST_NearestValue(e.rast, ab.location)) as elev_m
FROM elevation e
WHERE ST_Intersects(ab.location, e.rast)
GROUP BY ab.location
) AS subtable ON TRUE,
(SELECT name, last(location, timestamp) AS location FROM receiver_beacons_{postfix} GROUP BY name) AS r
WHERE ab.receiver_name = r.name;
DROP TABLE IF EXISTS "aircraft_beacons_{postfix}";
ALTER TABLE "aircraft_beacons_{postfix}_temp" RENAME TO "aircraft_beacons_{postfix}";
""".format(postfix=postfix))
def export_to_path(postfix, path):
def export_to_path(path):
connection = db.engine.raw_connection()
cursor = connection.cursor()
aircraft_beacons_file = os.path.join(path, "aircraft_beacons_{postfix}.csv.gz".format(postfix=postfix))
aircraft_beacons_file = os.path.join(path, "sender_positions.csv.gz")
with gzip.open(aircraft_beacons_file, "wt", encoding="utf-8") as gzip_file:
cursor.copy_expert("COPY ({}) TO STDOUT WITH (DELIMITER ',', FORMAT CSV, HEADER, ENCODING 'UTF-8');".format("SELECT * FROM aircraft_beacons_{postfix}".format(postfix=postfix)), gzip_file)
receiver_beacons_file = os.path.join(path, "receiver_beacons_{postfix}.csv.gz".format(postfix=postfix))
with gzip.open(receiver_beacons_file, "wt") as gzip_file:
cursor.copy_expert("COPY ({}) TO STDOUT WITH (DELIMITER ',', FORMAT CSV, HEADER, ENCODING 'UTF-8');".format("SELECT * FROM receiver_beacons_{postfix}".format(postfix=postfix)), gzip_file)
cursor.copy_expert("COPY ({}) TO STDOUT WITH (DELIMITER ',', FORMAT CSV, HEADER, ENCODING 'UTF-8');".format("SELECT * FROM sender_positions"), gzip_file)

1
app/main/__init__.py
Wyświetl plik

@ -3,3 +3,4 @@ from flask import Blueprint
bp = Blueprint("main", __name__)
import app.main.routes
import app.main.jinja_filters

64
app/main/jinja_filters.py 100644
Wyświetl plik

@ -0,0 +1,64 @@
from app.main import bp
from app.model import Airport, Sender, Receiver
from flask import url_for
import time
import datetime
import math
@bp.app_template_filter()
def timestamp_to_status(timestamp):
if datetime.datetime.utcnow() - timestamp < datetime.timedelta(minutes=10):
return 'OK'
elif datetime.datetime.utcnow() - timestamp < datetime.timedelta(hours=1):
return '<b>?</b>'
else:
return '<b>OFFLINE</b>'
@bp.app_template_filter()
def to_html_link(obj):
if isinstance(obj, Airport):
airport = obj
return f"""<a href="{url_for('main.airport_detail', airport_id=airport.id)}">{airport.name}</a>"""
elif isinstance(obj, Sender):
sender = obj
if len(sender.infos) > 0 and len(sender.infos[0].registration) > 0:
return f"""<a href="{url_for('main.sender_detail', sender_id=sender.id)}">{sender.infos[0].registration}</a>"""
elif sender.address:
return f"""<a href="{url_for('main.sender_detail', sender_id=sender.id)}">[{sender.address}]</a>"""
else:
return f"""<a href="{url_for('main.sender_detail', sender_id=sender.id)}">[{sender.name}]</a>"""
elif isinstance(obj, Receiver):
receiver = obj
return f"""<a href="{url_for('main.receiver_detail', receiver_id=receiver.id)}">{receiver.name}</a>"""
elif obj is None:
return "-"
else:
raise NotImplementedError("cant apply filter 'to_html_link' to object {type(obj)}")
@bp.app_template_filter()
def to_ordinal(rad):
deg = math.degrees(rad)
if deg >= 337.5 or deg < 22.5:
return "N"
elif deg >= 22.5 and deg < 67.5:
return "NW"
elif deg >= 67.5 and deg < 112.5:
return "W"
elif deg >= 112.5 and deg < 157.5:
return "SW"
elif deg >= 157.5 and deg < 202.5:
return "S"
elif deg >= 202.5 and deg < 247.5:
return "SE"
elif deg >= 247.5 and deg < 292.5:
return "E"
elif deg >= 292.5 and deg < 337.5:
return "NE"

87
app/main/live_routes.py
Wyświetl plik

@ -1,87 +0,0 @@
from flask import request, render_template, current_app
from flask_cors import cross_origin
from app.backend.liveglidernet import rec, lxml
from app.main import bp
@bp.route("/live.html")
@cross_origin()
def live():
return render_template("ogn_live.html", host=request.host)
@bp.route("/rec.php")
def rec_php():
a = request.args.get("a")
z = request.args.get("z")
xml = rec()
resp = current_app.make_response(xml)
resp.mimetype = "text/xml"
return resp
@bp.route("/lxml.php")
def lxml_php():
a = request.args.get("a")
b = request.args.get("b")
c = request.args.get("c")
d = request.args.get("d")
e = request.args.get("e")
z = request.args.get("z")
xml = lxml()
resp = current_app.make_response(xml)
resp.mimetype = "text/xml"
return resp
@bp.route("/pict/<filename>")
def pict(filename):
return current_app.send_static_file("ognlive/pict/" + filename)
@bp.route("/favicon.gif")
def favicon_gif():
return current_app.send_static_file("ognlive/pict/favicon.gif")
@bp.route("/horizZoomControl.js")
def horizZoomControl_js():
return current_app.send_static_file("ognlive/horizZoomControl.js")
@bp.route("/barogram.js")
def barogram_js():
return current_app.send_static_file("ognlive/barogram.js")
@bp.route("/util.js")
def util_js():
return current_app.send_static_file("ognlive/util.js")
@bp.route("/ogn.js")
def ogn_js():
return current_app.send_static_file("ognlive/ogn.js")
@bp.route("/ol.js")
def ol_js():
return current_app.send_static_file("ognlive/ol.js")
@bp.route("/osm.js")
def osm_js():
return current_app.send_static_file("ognlive/osm.js")
@bp.route("/ol.css")
def ol_css():
return current_app.send_static_file("ognlive/ol.css")
@bp.route("/osm.css")
def osm_css():
return current_app.send_static_file("ognlive/osm.css")

44
app/main/matplotlib_service.py 100644
Wyświetl plik

@ -0,0 +1,44 @@
from app import db
from app.model import SenderDirectionStatistic
import random
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.figure import Figure
def create_range_figure2(sender_id):
fig = Figure()
axis = fig.add_subplot(1, 1, 1)
xs = range(100)
ys = [random.randint(1, 50) for x in xs]
axis.plot(xs, ys)
return fig
def create_range_figure(sender_id):
sds = db.session.query(SenderDirectionStatistic) \
.filter(SenderDirectionStatistic.sender_id == sender_id) \
.order_by(SenderDirectionStatistic.directions_count.desc()) \
.limit(1) \
.one()
fig = Figure()
direction_data = sds.direction_data
max_range = max([r['max_range'] / 1000.0 for r in direction_data])
theta = np.array([i['direction'] / 180 * np.pi for i in direction_data])
radii = np.array([i['max_range'] / 1000 if i['max_range'] > 0 else 0 for i in direction_data])
width = np.array([13 / 180 * np.pi for i in direction_data])
colors = plt.cm.viridis(radii / max_range)
ax = fig.add_subplot(111, projection='polar')
ax.bar(theta, radii, width=width, bottom=0.0, color=colors, edgecolor='b', alpha=0.5)
#ax.set_rticks([0, 25, 50, 75, 100, 125, 150])
ax.set_theta_zero_location("N")
ax.set_theta_direction(-1)
fig.suptitle(f"Range between sender '{sds.sender.name}' and receiver '{sds.receiver.name}'")
return fig

184
app/main/routes.py
Wyświetl plik

@ -1,12 +1,13 @@
import datetime
from datetime import date, time, datetime
from flask import request, render_template, send_file
from app import db
from app import cache
from app.model import Airport, Country, Device, Logbook, Receiver, ReceiverStats
from app.model import Airport, Country, Sender, SenderInfo, TakeoffLanding, Logbook, Receiver, SenderPosition, RelationStatistic, ReceiverStatistic, SenderStatistic
from app.main import bp
from app.main.matplotlib_service import create_range_figure
@cache.cached(key_prefix="countries_in_receivers")
@ -16,28 +17,26 @@ def get_countries_in_receivers():
return [{"iso2": country[0]} for country in query.all()]
@cache.cached(key_prefix="countries_in_logbook")
def get_countries_in_logbook():
query = db.session.query(Country.iso2).filter(Country.iso2 == Airport.country_code).filter(Logbook.takeoff_airport_id == Airport.id).order_by(Country.iso2).distinct(Country.iso2)
@cache.cached(key_prefix="countries_in_takeoff_landings")
def get_used_countries():
query = db.session.query(Country.iso2).filter(Country.gid == TakeoffLanding.country_id).order_by(Country.iso2).distinct(Country.iso2)
return [{"iso2": country[0]} for country in query.all()]
@cache.memoize()
def get_airports_in_country(sel_country):
query = db.session.query(Airport.id, Airport.name).filter(Airport.country_code == sel_country).filter(Logbook.takeoff_airport_id == Airport.id).order_by(Airport.name).distinct(Airport.name)
return [{"id": airport[0], "name": airport[1]} for airport in query.all()]
def get_used_airports_by_country(sel_country):
query = db.session.query(Airport).filter(Airport.country_code == sel_country).filter(TakeoffLanding.airport_id == Airport.id).filter(TakeoffLanding.country_id == Country.gid).order_by(Airport.name).distinct(Airport.name)
return [used_airport for used_airport in query]
@cache.memoize()
def get_dates_for_airport(sel_airport):
query = (
db.session.query(db.func.date(Logbook.reftime), db.func.count(Logbook.id).label("logbook_count"))
db.session.query(db.func.date(Logbook.reference_timestamp), db.func.count(Logbook.id).label("logbook_count"))
.filter(Airport.id == sel_airport)
.filter(db.or_(Airport.id == Logbook.takeoff_airport_id, Airport.id == Logbook.landing_airport_id))
.group_by(db.func.date(Logbook.reftime))
.order_by(db.func.date(Logbook.reftime).desc())
.group_by(db.func.date(Logbook.reference_timestamp))
.order_by(db.func.date(Logbook.reference_timestamp).desc())
)
return [{"date": date, "logbook_count": logbook_count} for (date, logbook_count) in query.all()]
@ -46,21 +45,56 @@ def get_dates_for_airport(sel_airport):
@bp.route("/")
@bp.route("/index.html")
def index():
return render_template("base.html")
today_beginning = datetime.combine(date.today(), time())
senders_today = db.session.query(db.func.count(Sender.id)).filter(Sender.lastseen >= today_beginning).one()[0]
receivers_today = db.session.query(db.func.count(Receiver.id)).filter(Receiver.lastseen >= today_beginning).one()[0]
takeoffs_today = db.session.query(db.func.count(TakeoffLanding.id)).filter(db.and_(TakeoffLanding.timestamp >= today_beginning, TakeoffLanding.is_takeoff is True)).one()[0]
landings_today = db.session.query(db.func.count(TakeoffLanding.id)).filter(db.and_(TakeoffLanding.timestamp >= today_beginning, TakeoffLanding.is_takeoff is False)).one()[0]
sender_positions_today = db.session.query(db.func.sum(ReceiverStatistic.messages_count)).filter(ReceiverStatistic.date == date.today()).one()[0]
sender_positions_total = db.session.query(db.func.sum(ReceiverStatistic.messages_count)).one()[0]
last_logbook_entries = db.session.query(Logbook).order_by(Logbook.reference_timestamp.desc()).limit(10)
return render_template(
"index.html",
senders_today=senders_today,
receivers_today=receivers_today,
takeoffs_today=takeoffs_today,
landings_today=landings_today,
sender_positions_today=sender_positions_today,
sender_positions_total=sender_positions_total,
logbook=last_logbook_entries)
@bp.route("/devices.html", methods=["GET", "POST"])
def devices():
devices = db.session.query(Device).order_by(Device.address)
return render_template("devices.html", devices=devices)
@bp.route("/senders.html", methods=["GET", "POST"])
def senders():
senders = db.session.query(Sender) \
.options(db.joinedload(Sender.infos)) \
.order_by(Sender.name)
return render_template("senders.html", senders=senders)
@bp.route("/device_detail.html", methods=["GET", "POST"])
def device_detail():
device_id = request.args.get("id")
device = db.session.query(Device).filter(Device.id == device_id).one()
@bp.route("/sender_detail.html", methods=["GET", "POST"])
def sender_detail():
sender_id = request.args.get("sender_id")
sender = db.session.query(Sender).filter(Sender.id == sender_id).one()
return render_template("device_detail.html", title="Device", device=device)
return render_template("sender_detail.html", title="Sender", sender=sender)
@bp.route("/range_view.png")
def range_view():
import io
from flask import Response
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
sender_id = request.args.get("sender_id")
fig = create_range_figure(sender_id)
output = io.BytesIO()
FigureCanvas(fig).print_png(output)
return Response(output.getvalue(), mimetype='image/png')
@bp.route("/receivers.html")
@ -71,41 +105,34 @@ def receivers():
# Get receiver selection list
if sel_country:
receivers = db.session.query(Receiver).filter(db.and_(Receiver.country_id == Country.gid, Country.iso2 == sel_country)).order_by(Receiver.name)
receivers = db.session.query(Receiver) \
.options(db.joinedload(Receiver.airport)) \
.filter(db.and_(Receiver.country_id == Country.gid, Country.iso2 == sel_country)) \
.order_by(Receiver.name)
else:
receivers = db.session.query(Receiver).order_by(Receiver.name)
receivers = db.session.query(Receiver) \
.options(db.joinedload(Receiver.airport)) \
.order_by(Receiver.name)
return render_template("receivers.html", title="Receivers", sel_country=sel_country, countries=countries, receivers=receivers)
@bp.route("/receiver_detail.html")
def receiver_detail():
sel_receiver_id = request.args.get("receiver_id")
receiver_id = request.args.get("receiver_id")
receiver = db.session.query(Receiver).filter(Receiver.id == sel_receiver_id).one()
airport = (
db.session.query(Airport)
.filter(
db.and_(
Receiver.id == sel_receiver_id,
db.func.st_contains(db.func.st_buffer(Receiver.location_wkt, 0.5), Airport.location_wkt),
db.func.st_distance_sphere(Airport.location_wkt, Receiver.location_wkt) < 1000,
)
)
.filter(Airport.style.in_((2, 4, 5)))
)
return render_template("receiver_detail.html", title="Receiver Detail", receiver=receiver, airport=airport.first())
receiver = db.session.query(Receiver).filter(Receiver.id == receiver_id).one()
return render_template("receiver_detail.html", title="Receiver Detail", receiver=receiver)
@bp.route("/airports.html", methods=["GET", "POST"])
def airports():
sel_country = request.args.get("country")
countries = get_countries_in_logbook()
countries = get_used_countries()
if sel_country:
airports = get_airports_in_country(sel_country)
airports = get_used_airports_by_country(sel_country)
else:
airports = []
@ -116,41 +143,41 @@ def airports():
@bp.route("/airport_detail.html")
def airport_detail():
sel_airport = request.args.get("airport")
sel_airport = request.args.get("airport_id")
airport = db.session.query(Airport).filter(Airport.id == sel_airport)
devices = db.session.query(Device).join(Logbook).filter(Logbook.takeoff_airport_id == sel_airport).order_by(Device.address)
senders = db.session.query(Sender).join(Logbook).filter(Logbook.takeoff_airport_id == sel_airport).order_by(Sender.name)
return render_template("airport_detail.html", title="Airport Detail", airport=airport.one(), devices=devices)
return render_template("airport_detail.html", title="Airport Detail", airport=airport.one(), senders=senders)
@bp.route("/logbook.html", methods=["GET", "POST"])
def logbook():
@bp.route("/logbooks.html", methods=["GET", "POST"])
def logbooks():
sel_country = request.args.get("country")
sel_airport = request.args.get("airport")
sel_airport_id = request.args.get("airport_id")
sel_date = request.args.get("date")
sel_device_id = request.args.get("device_id")
sel_sender_id = request.args.get("sender_id")
countries = get_countries_in_logbook()
countries = get_used_countries()
if sel_country:
airports = get_airports_in_country(sel_country)
airports = get_used_airports_by_country(sel_country)
else:
airports = []
if sel_airport:
sel_airport = int(sel_airport)
if sel_airport not in [airport["id"] for airport in airports]:
sel_airport = None
if sel_airport_id:
sel_airport_id = int(sel_airport_id)
if sel_airport_id not in [airport.id for airport in airports]:
sel_airport_id = None
sel_date = None
dates = get_dates_for_airport(sel_airport)
dates = get_dates_for_airport(sel_airport_id)
else:
dates = []
if sel_date:
sel_date = datetime.datetime.strptime(sel_date, "%Y-%m-%d").date()
sel_date = datetime.strptime(sel_date, "%Y-%m-%d").date()
if sel_date not in [entry["date"] for entry in dates]:
sel_date = dates[0]["date"]
elif len(dates) > 0:
@ -158,21 +185,21 @@ def logbook():
# Get Logbook
filters = []
if sel_airport:
filters.append(db.or_(Logbook.takeoff_airport_id == sel_airport, Logbook.landing_airport_id == sel_airport))
if sel_airport_id:
filters.append(db.or_(Logbook.takeoff_airport_id == sel_airport_id, Logbook.landing_airport_id == sel_airport_id))
if sel_date:
filters.append(db.func.date(Logbook.reftime) == sel_date)
filters.append(db.func.date(Logbook.reference_timestamp) == sel_date)
if sel_device_id:
filters.append(Logbook.device_id == sel_device_id)
if sel_sender_id:
filters.append(Logbook.sender_id == sel_sender_id)
if len(filters) > 0:
logbook = db.session.query(Logbook).filter(*filters).order_by(Logbook.reftime)
logbooks = db.session.query(Logbook).filter(*filters).order_by(Logbook.reference_timestamp).limit(100)
else:
logbook = None
logbooks = None
return render_template("logbook.html", title="Logbook", sel_country=sel_country, countries=countries, sel_airport=sel_airport, airports=airports, sel_date=sel_date, dates=dates, logbook=logbook)
return render_template("logbooks.html", title="Logbook", sel_country=sel_country, countries=countries, sel_airport_id=sel_airport_id, airports=airports, sel_date=sel_date, dates=dates, logbooks=logbooks)
@bp.route("/download.html")
@ -186,12 +213,27 @@ def download_flight():
return send_file(buffer, as_attachment=True, attachment_filename="wtf.igc", mimetype="text/plain")
@bp.route("/statistics.html")
def statistics():
@bp.route("/sender_ranking.html")
def sender_ranking():
sender_statistics = db.session.query(SenderStatistic) \
.filter(db.and_(SenderStatistic.date == date.today(), SenderStatistic.is_trustworthy is True)) \
.order_by(SenderStatistic.max_distance.desc()) \
.all()
today = datetime.date.today()
today = datetime.date(2018, 7, 31)
return render_template(
"sender_ranking.html",
title="Sender Ranking",
ranking=sender_statistics)
receiverstats = db.session.query(ReceiverStats).filter(ReceiverStats.date == today)
return render_template("statistics.html", title="Receiver Statistics", receiverstats=receiverstats)
@bp.route("/receiver_ranking.html")
def receiver_ranking():
receiver_statistics = db.session.query(ReceiverStatistic) \
.filter(db.and_(ReceiverStatistic.date == date.today(), ReceiverStatistic.is_trustworthy is True)) \
.order_by(ReceiverStatistic.max_distance.desc()) \
.all()
return render_template(
"receiver_ranking.html",
title="Receiver Ranking",
ranking=receiver_statistics)

25
app/model/__init__.py
Wyświetl plik

@ -1,21 +1,22 @@
# flake8: noqa
from .aircraft_type import AircraftType
from .beacon import Beacon
from .country import Country
from .country_stats import CountryStats
from .device import Device
from .device_info import DeviceInfo
from .device_info_origin import DeviceInfoOrigin
from .device_stats import DeviceStats
from .aircraft_beacon import AircraftBeacon
from .receiver_beacon import ReceiverBeacon
from .sender import Sender
from .sender_info_origin import SenderInfoOrigin
from .sender_info import SenderInfo
from .sender_position import SenderPosition
from .receiver_position import ReceiverPosition
from .receiver_status import ReceiverStatus
from .receiver import Receiver
from .receiver_stats import ReceiverStats
from .takeoff_landing import TakeoffLanding
from .airport import Airport
from .logbook import Logbook
from .receiver_coverage import ReceiverCoverage
from .relation_stats import RelationStats
from .flights2d import Flight2D
from .geo import Location
from .relation_statistic import RelationStatistic
from .coverage_statistic import CoverageStatistic
from .sender_statistic import SenderStatistic
from .receiver_statistic import ReceiverStatistic
from .sender_position_statistic import SenderPositionStatistic
from .sender_direction_statistic import SenderDirectionStatistic

130
app/model/aircraft_beacon.py
Wyświetl plik

@ -1,130 +0,0 @@
from sqlalchemy.sql import func
from app import db
from .beacon import Beacon
from .aircraft_type import AircraftType
class AircraftBeacon(Beacon):
__tablename__ = "aircraft_beacons"
# Flarm specific data
address_type = db.Column(db.SmallInteger)
aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN)
stealth = db.Column(db.Boolean)
address = db.Column(db.String)
climb_rate = db.Column(db.Float(precision=2))
turn_rate = db.Column(db.Float(precision=2))
signal_quality = db.Column(db.Float(precision=2))
error_count = db.Column(db.SmallInteger)
frequency_offset = db.Column(db.Float(precision=2))
gps_quality_horizontal = db.Column(db.SmallInteger)
gps_quality_vertical = db.Column(db.SmallInteger)
software_version = db.Column(db.Float(precision=2))
hardware_version = db.Column(db.SmallInteger)
real_address = db.Column(db.String(6))
signal_power = db.Column(db.Float(precision=2))
proximity = None
# Calculated values
distance = db.Column(db.Float(precision=2))
radial = db.Column(db.SmallInteger)
quality = db.Column(db.Float(precision=2)) # signal quality normalized to 10km
location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars)
location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool
agl = db.Column(db.Float(precision=2))
def __repr__(self):
return "<AircraftBeacon %s: %s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s>" % (
self.address_type,
self.aircraft_type,
self.stealth,
self.address,
self.climb_rate,
self.turn_rate,
self.signal_quality,
self.error_count,
self.frequency_offset,
self.gps_quality_horizontal,
self.gps_quality_vertical,
self.software_version,
self.hardware_version,
self.real_address,
self.signal_power,
self.distance,
self.radial,
self.quality,
self.location_mgrs,
self.location_mgrs_short,
)
@classmethod
def get_columns(self):
return [
"location",
"altitude",
"name",
"dstcall",
"relay",
"receiver_name",
"timestamp",
"track",
"ground_speed",
# 'raw_message',
# 'reference_timestamp',
"address_type",
"aircraft_type",
"stealth",
"address",
"climb_rate",
"turn_rate",
"signal_quality",
"error_count",
"frequency_offset",
"gps_quality_horizontal",
"gps_quality_vertical",
"software_version",
"hardware_version",
"real_address",
"signal_power",
"distance",
"radial",
"quality",
"location_mgrs",
"location_mgrs_short",
]
def get_values(self):
return [
self.location_wkt,
int(self.altitude) if self.altitude else None,
self.name,
self.dstcall,
self.relay,
self.receiver_name,
self.timestamp,
self.track,
self.ground_speed,
# self.raw_message,
# self.reference_timestamp,
self.address_type,
self.aircraft_type,
self.stealth,
self.address,
self.climb_rate,
self.turn_rate,
self.signal_quality,
self.error_count,
self.frequency_offset,
self.gps_quality_horizontal,
self.gps_quality_vertical,
self.software_version,
self.hardware_version,
self.real_address,
self.signal_power,
self.distance,
self.radial,
self.quality,
self.location_mgrs,
self.location_mgrs_short,
]

45
app/model/beacon.py
Wyświetl plik

@ -1,45 +0,0 @@
from geoalchemy2.shape import to_shape
from geoalchemy2.types import Geometry
from sqlalchemy.ext.declarative import AbstractConcreteBase
from sqlalchemy.ext.hybrid import hybrid_property
from .geo import Location
from app import db
class Beacon(AbstractConcreteBase, db.Model):
# APRS data
location_wkt = db.Column("location", Geometry("POINT", srid=4326))
altitude = db.Column(db.Float(precision=2))
name = db.Column(db.String, primary_key=True, nullable=True)
dstcall = db.Column(db.String)
relay = db.Column(db.String)
receiver_name = db.Column(db.String(9), primary_key=True, nullable=True)
timestamp = db.Column(db.DateTime, primary_key=True, nullable=True)
symboltable = None
symbolcode = None
track = db.Column(db.SmallInteger)
ground_speed = db.Column(db.Float(precision=2))
comment = None
# Type information
beacon_type = None
aprs_type = None
# Debug information
raw_message = None
reference_timestamp = None
@hybrid_property
def location(self):
if self.location_wkt is None:
return None
coords = to_shape(self.location_wkt)
return Location(lat=coords.y, lon=coords.x)
@location.expression
def location(cls):
return cls.location_wkt

17
app/model/country_stats.py
Wyświetl plik

@ -1,17 +0,0 @@
from app import db
class CountryStats(db.Model):
__tablename__ = "country_stats"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
# Static data
aircraft_beacon_count = db.Column(db.Integer)
device_count = db.Column(db.Integer)
# Relations
country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="SET NULL"), index=True)
country = db.relationship("Country", foreign_keys=[country_id], backref=db.backref("stats", order_by="CountryStats.date.asc()"))

25
app/model/coverage_statistic.py 100644
Wyświetl plik

@ -0,0 +1,25 @@
from app import db
class CoverageStatistic(db.Model):
__tablename__ = "coverage_statistics"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
location_mgrs_short = db.Column(db.String(9))
is_trustworthy = db.Column(db.Boolean)
messages_count = db.Column(db.Integer)
max_distance = db.Column(db.Float(precision=2))
max_normalized_quality = db.Column(db.Float(precision=2))
coverages_count = db.Column(db.Integer)
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True)
sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("coverage_stats", order_by=date))
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("coverage_stats", order_by=date))
__table_args__ = (db.Index('idx_coverage_statistics_uc', 'date', 'location_mgrs_short', 'sender_id', 'receiver_id', 'is_trustworthy', unique=True), )

8
app/model/device_info_origin.py
Wyświetl plik

@ -1,8 +0,0 @@
import enum
class DeviceInfoOrigin(enum.Enum):
UNKNOWN = 0
OGN_DDB = 1
FLARMNET = 2
USER_DEFINED = 3

52
app/model/device_stats.py
Wyświetl plik

@ -1,52 +0,0 @@
from app import db
from .aircraft_type import AircraftType
class DeviceStats(db.Model):
__tablename__ = "device_stats"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
# Static data
name = db.Column(db.String)
firstseen = db.Column(db.DateTime)
lastseen = db.Column(db.DateTime)
aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN)
stealth = db.Column(db.Boolean)
software_version = db.Column(db.Float(precision=2))
hardware_version = db.Column(db.SmallInteger)
real_address = db.Column(db.String(6))
# Statistic data
max_altitude = db.Column(db.Float(precision=2))
receiver_count = db.Column(db.SmallInteger)
aircraft_beacon_count = db.Column(db.Integer)
jumps = db.Column(db.SmallInteger)
ambiguous = db.Column(db.Boolean)
quality = db.Column(db.Float(precision=2))
# Relation statistic data
quality_offset = db.Column(db.Float(precision=2))
# Ranking data
max_altitude_ranking_worldwide = db.Column(db.Integer)
max_altitude_ranking_country = db.Column(db.Integer)
receiver_count_ranking_worldwide = db.Column(db.Integer)
receiver_count_ranking_country = db.Column(db.Integer)
aircraft_beacon_count_ranking_worldwide = db.Column(db.Integer)
aircraft_beacon_count_ranking_country = db.Column(db.Integer)
quality_ranking_worldwide = db.Column(db.Integer)
quality_ranking_country = db.Column(db.Integer)
# Relations
device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), index=True)
device = db.relationship("Device", foreign_keys=[device_id], backref=db.backref("stats", order_by="DeviceStats.date.asc()"))
def __repr__(self):
return "<DeviceStats: %s,%s,%s,%s>" % (self.date, self.receiver_count, self.aircraft_beacon_count, self.max_altitude)
db.Index("ix_device_stats_date_device_id", DeviceStats.date, DeviceStats.device_id)

24
app/model/flights2d.py
Wyświetl plik

@ -1,24 +0,0 @@
from geoalchemy2.types import Geometry
from app import db
class Flight2D(db.Model):
__tablename__ = "flights2d"
date = db.Column(db.Date, primary_key=True)
flight_type = db.Column(db.SmallInteger, primary_key=True)
path_wkt = db.Column("path", Geometry("MULTILINESTRING", srid=4326))
path_simple_wkt = db.Column("path_simple", Geometry("MULTILINESTRING", srid=4326)) # this is the path simplified with ST_Simplify(path, 0.0001)
# Relations
device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), primary_key=True)
device = db.relationship("Device", foreign_keys=[device_id], backref="flights2d")
def __repr__(self):
return "<Flight %s: %s,%s>" % (self.date, self.path_wkt, self.path_simple_wkt)
db.Index("ix_flights2d_date_device_id", Flight2D.date, Flight2D.device_id)
# db.Index('ix_flights2d_date_path', Flight2D.date, Flight2D.path_wkt) --> CREATE INDEX ix_flights2d_date_path ON flights2d USING GIST("date", path)

39
app/model/logbook.py
Wyświetl plik

@ -1,14 +1,13 @@
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.sql import null, case
from app import db
class Logbook(db.Model):
__tablename__ = "logbook"
__tablename__ = "logbooks"
id = db.Column(db.Integer, primary_key=True)
reftime = db.Column(db.DateTime, index=True)
takeoff_timestamp = db.Column(db.DateTime)
takeoff_track = db.Column(db.SmallInteger)
landing_timestamp = db.Column(db.DateTime)
@ -16,14 +15,20 @@ class Logbook(db.Model):
max_altitude = db.Column(db.Float(precision=2))
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True)
sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("logbook_entries", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc()))
takeoff_airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="CASCADE"), index=True)
takeoff_airport = db.relationship("Airport", foreign_keys=[takeoff_airport_id])
takeoff_airport = db.relationship("Airport", foreign_keys=[takeoff_airport_id], backref=db.backref("logbook_entries_takeoff", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc()))
takeoff_country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="CASCADE"), index=True)
takeoff_country = db.relationship("Country", foreign_keys=[takeoff_country_id], backref=db.backref("logbook_entries_takeoff", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc()))
landing_airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="CASCADE"), index=True)
landing_airport = db.relationship("Airport", foreign_keys=[landing_airport_id])
landing_airport = db.relationship("Airport", foreign_keys=[landing_airport_id], backref=db.backref("logbook_entries_landing", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc()))
device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="CASCADE"), index=True)
device = db.relationship("Device", foreign_keys=[device_id], backref=db.backref("logbook", order_by="Logbook.reftime"))
landing_country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="CASCADE"), index=True)
landing_country = db.relationship("Country", foreign_keys=[landing_country_id], backref=db.backref("logbook_entries_landing", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc()))
@hybrid_property
def duration(self):
@ -31,4 +36,22 @@ class Logbook(db.Model):
@duration.expression
def duration(cls):
return case({False: None, True: cls.landing_timestamp - cls.takeoff_timestamp}, cls.landing_timestamp != null() and cls.takeoff_timestamp != null())
return db.case({False: None, True: cls.landing_timestamp - cls.takeoff_timestamp}, cls.landing_timestamp != db.null() and cls.takeoff_timestamp != db.null())
@hybrid_property
def reference_timestamp(self):
return self.takeoff_timestamp if self.takeoff_timestamp is not None else self.landing_timestamp
@reference_timestamp.expression
def reference_timestamp(cls):
return db.case({True: cls.takeoff_timestamp, False: cls.landing_timestamp}, cls.takeoff_timestamp != db.null())
#__table_args__ = (db.Index('idx_logbook_reference_timestamp', db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null())),)
# FIXME: does not work...
# FIXME: this does not throw an error as the __table_args__ above, but there is no index created
#_wrapped_case = f"({db.case(whens={True: Logbook.takeoff_timestamp, False: Logbook.landing_timestamp}, value=Logbook.takeoff_timestamp != db.null())})"
#Index("idx_logbook_reference_timestamp", _wrapped_case)
# TODO:
# so execute manually: CREATE INDEX IF NOT EXISTS idx_logbook_reference_timestamp ON logbooks ((CASE takeoff_timestamp IS NULL WHEN true THEN takeoff_timestamp WHEN false THEN landing_timestamp END));

25
app/model/receiver.py
Wyświetl plik

@ -5,25 +5,37 @@ from .geo import Location
from app import db
from .airport import Airport
class Receiver(db.Model):
__tablename__ = "receivers"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(9))
location_wkt = db.Column("location", Geometry("POINT", srid=4326))
altitude = db.Column(db.Float(precision=2))
name = db.Column(db.String(9), index=True)
firstseen = db.Column(db.DateTime, index=True)
lastseen = db.Column(db.DateTime, index=True)
timestamp = db.Column(db.DateTime, index=True)
version = db.Column(db.String)
platform = db.Column(db.String)
cpu_temp = db.Column(db.Float(precision=2))
rec_input_noise = db.Column(db.Float(precision=2))
agl = db.Column(db.Float(precision=2))
# Relations
country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="SET NULL"), index=True)
country = db.relationship("Country", foreign_keys=[country_id], backref=db.backref("receivers", order_by="Receiver.name.asc()"))
airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="CASCADE"), index=True)
airport = db.relationship("Airport", foreign_keys=[airport_id], backref=db.backref("receivers", order_by="Receiver.name.asc()"))
__table_args__ = (db.Index('idx_receivers_name_uc', 'name', unique=True), )
@property
def location(self):
if self.location_wkt is None:
@ -31,3 +43,14 @@ class Receiver(db.Model):
coords = to_shape(self.location_wkt)
return Location(lat=coords.y, lon=coords.x)
def airports_nearby(self):
query = (
db.session.query(Airport, db.func.st_distance_sphere(self.location_wkt, Airport.location_wkt), db.func.st_azimuth(self.location_wkt, Airport.location_wkt))
.filter(db.func.st_contains(db.func.st_buffer(Airport.location_wkt, 1), self.location_wkt))
.filter(Airport.style.in_((2, 4, 5)))
.order_by(db.func.st_distance_sphere(self.location_wkt, Airport.location_wkt).asc())
.limit(5)
)
airports = [(airport, distance, azimuth) for airport, distance, azimuth in query]
return airports

45
app/model/receiver_beacon.py
Wyświetl plik

@ -1,45 +0,0 @@
from .beacon import Beacon
class ReceiverBeacon(Beacon):
__tablename__ = "receiver_beacons"
# disable irrelevant aprs fields
relay = None
track = None
ground_speed = None
def __repr__(self):
return "<ReceiverBeacon %s: %s,%s,%s,%s,%s>" % (
self.name,
self.location,
self.altitude,
self.dstcall,
self.receiver_name,
self.timestamp,
)
@classmethod
def get_columns(self):
return [
"location",
"altitude",
"name",
"dstcall",
"receiver_name",
"timestamp",
# 'raw_message',
# 'reference_timestamp',
]
def get_values(self):
return [
self.location_wkt,
int(self.altitude) if self.altitude else None,
self.name,
self.dstcall,
self.receiver_name,
self.timestamp,
# self.raw_message,
# self.reference_timestamp,
]

23
app/model/receiver_coverage.py
Wyświetl plik

@ -1,23 +0,0 @@
from app import db
class ReceiverCoverage(db.Model):
__tablename__ = "receiver_coverages"
location_mgrs_short = db.Column(db.String(9), primary_key=True)
date = db.Column(db.Date, primary_key=True)
max_signal_quality = db.Column(db.Float)
max_altitude = db.Column(db.Float(precision=2))
min_altitude = db.Column(db.Float(precision=2))
aircraft_beacon_count = db.Column(db.Integer)
device_count = db.Column(db.SmallInteger)
# Relations
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="SET NULL"), primary_key=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("receiver_coverages", order_by="ReceiverCoverage.date.asc()"))
db.Index("ix_receiver_coverages_date_receiver_id", ReceiverCoverage.date, ReceiverCoverage.receiver_id)
db.Index("ix_receiver_coverages_receiver_id_date", ReceiverCoverage.receiver_id, ReceiverCoverage.date)

39
app/model/receiver_position.py 100644
Wyświetl plik

@ -0,0 +1,39 @@
from geoalchemy2.types import Geometry
from app import db
class ReceiverPosition(db.Model):
__tablename__ = "receiver_positions"
reference_timestamp = db.Column(db.DateTime, primary_key=True)
# APRS data
name = db.Column(db.String)
dstcall = db.Column(db.String)
#relay = db.Column(db.String)
receiver_name = db.Column(db.String(9))
timestamp = db.Column(db.DateTime)
location = db.Column("location", Geometry("POINT", srid=4326))
symboltable = None
symbolcode = None
#track = db.Column(db.SmallInteger)
#ground_speed = db.Column(db.Float(precision=2))
altitude = db.Column(db.Float(precision=2))
comment = None
# Type information
beacon_type = None
aprs_type = None
# Debug information
raw_message = None
# Receiver specific data
user_comment = None
# Calculated values (from this software)
location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars)
location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool
agl = db.Column(db.Float(precision=2))

22
app/model/receiver_statistic.py 100644
Wyświetl plik

@ -0,0 +1,22 @@
from app import db
class ReceiverStatistic(db.Model):
__tablename__ = "receiver_statistics"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
is_trustworthy = db.Column(db.Boolean)
max_distance = db.Column(db.Float(precision=2))
max_normalized_quality = db.Column(db.Float(precision=2))
messages_count = db.Column(db.Integer)
coverages_count = db.Column(db.Integer)
senders_count = db.Column(db.Integer)
# Relations
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("statistics", order_by=date.desc()))
__table_args__ = (db.Index('idx_receiver_statistics_uc', 'date', 'receiver_id', 'is_trustworthy', unique=True), )

41
app/model/receiver_stats.py
Wyświetl plik

@ -1,41 +0,0 @@
from geoalchemy2.types import Geometry
from app import db
class ReceiverStats(db.Model):
__tablename__ = "receiver_stats"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
# Static data
firstseen = db.Column(db.DateTime, index=True)
lastseen = db.Column(db.DateTime, index=True)
location_wkt = db.Column("location", Geometry("POINT", srid=4326))
altitude = db.Column(db.Float(precision=2))
version = db.Column(db.String)
platform = db.Column(db.String)
# Statistic data
aircraft_beacon_count = db.Column(db.Integer)
aircraft_count = db.Column(db.SmallInteger)
max_distance = db.Column(db.Float)
quality = db.Column(db.Float(precision=2))
# Relation statistic data
quality_offset = db.Column(db.Float(precision=2))
# Ranking data
aircraft_beacon_count_ranking = db.Column(db.SmallInteger)
aircraft_count_ranking = db.Column(db.SmallInteger)
max_distance_ranking = db.Column(db.SmallInteger)
quality_ranking = db.Column(db.Integer)
# Relations
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="SET NULL"), index=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("stats", order_by="ReceiverStats.date.asc()"))
db.Index("ix_receiver_stats_date_receiver_id", ReceiverStats.date, ReceiverStats.receiver_id)

48
app/model/receiver_status.py 100644
Wyświetl plik

@ -0,0 +1,48 @@
from app import db
class ReceiverStatus(db.Model):
__tablename__ = "receiver_statuses"
reference_timestamp = db.Column(db.DateTime, primary_key=True)
# APRS data
name = db.Column(db.String)
dstcall = db.Column(db.String)
receiver_name = db.Column(db.String(9))
timestamp = db.Column(db.DateTime)
# Type information
beacon_type = None
aprs_type = None
# Debug information
raw_message = None
# Receiver specific data
version = db.Column(db.String)
platform = db.Column(db.String)
cpu_load = None
free_ram = None
total_ram = None
ntp_error = None
rt_crystal_correction = None
voltage = None
amperage = None
cpu_temp = db.Column(db.Float(precision=2))
senders_visible = None
senders_total = None
rec_crystal_correction = None
rec_crystal_correction_fine = None
rec_input_noise = db.Column(db.Float(precision=2))
senders_signal = None
senders_messages = None
good_senders_signal = None
good_senders = None
good_and_bad_senders = None
# Calculated values (from this software)
location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars)
location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool
agl = db.Column(db.Float(precision=2))

23
app/model/relation_statistic.py 100644
Wyświetl plik

@ -0,0 +1,23 @@
from app import db
class RelationStatistic(db.Model):
__tablename__ = "relation_statistics"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
is_trustworthy = db.Column(db.Boolean)
max_distance = db.Column(db.Float(precision=2))
max_normalized_quality = db.Column(db.Float(precision=2))
messages_count = db.Column(db.Integer)
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True)
sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("relation_stats", order_by=date))
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("relation_stats", order_by=date))
__table_args__ = (db.Index('idx_relation_statistics_uc', 'date', 'sender_id', 'receiver_id', 'is_trustworthy', unique=True), )

26
app/model/relation_stats.py
Wyświetl plik

@ -1,26 +0,0 @@
from app import db
class RelationStats(db.Model):
__tablename__ = "relation_stats"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
# Statistic data
quality = db.Column(db.Float(precision=2))
beacon_count = db.Column(db.Integer)
# Relations
device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), index=True)
device = db.relationship("Device", foreign_keys=[device_id], backref="relation_stats")
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="SET NULL"), index=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref="relation_stats")
def __repr__(self):
return "<RelationStats: %s,%s,%s>" % (self.date, self.quality, self.beacon_count)
db.Index("ix_relation_stats_date_device_id", RelationStats.date, RelationStats.device_id, RelationStats.receiver_id)
db.Index("ix_relation_stats_date_receiver_id", RelationStats.date, RelationStats.receiver_id, RelationStats.device_id)

29
app/model/device.py → app/model/sender.py
Wyświetl plik

@ -3,18 +3,16 @@ import datetime
from sqlalchemy.ext.hybrid import hybrid_property
from app import db
from .device_info import DeviceInfo
from app.model.aircraft_type import AircraftType
class Device(db.Model):
__tablename__ = "devices"
class Sender(db.Model):
__tablename__ = "senders"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
name = db.Column(db.String, index=True)
# address = db.Column(db.String(6), index=True)
address = db.Column(db.String, index=True)
address = db.Column(db.String(6), index=True)
firstseen = db.Column(db.DateTime, index=True)
lastseen = db.Column(db.DateTime, index=True)
aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN)
@ -23,21 +21,16 @@ class Device(db.Model):
hardware_version = db.Column(db.SmallInteger)
real_address = db.Column(db.String(6))
__table_args__ = (db.Index('idx_senders_name_uc', 'name', unique=True), )
def __repr__(self):
return "<Device: %s,%s,%s,%s,%s,%s>" % (self.address, self.aircraft_type, self.stealth, self.software_version, self.hardware_version, self.real_address)
@hybrid_property
def info(self):
query = db.session.query(DeviceInfo).filter(DeviceInfo.address == self.address).order_by(DeviceInfo.address_origin)
return query.first()
def get_infos(self):
query = db.session.query(DeviceInfo).filter(DeviceInfo.address == self.address).order_by(DeviceInfo.address_origin)
return [info for info in query.all()]
return "<Sender: %s,%s,%s,%s,%s,%s>" % (self.address, self.aircraft_type, self.stealth, self.software_version, self.hardware_version, self.real_address)
EXPIRY_DATES = {
7.01: datetime.date(2022, 2, 28),
7.0: datetime.date(2021, 10, 31),
6.83: datetime.date(2021, 10, 31),
6.82: datetime.date(2021, 5, 31),
6.81: datetime.date(2021, 1, 31),
6.80: datetime.date(2021, 1, 31),
6.67: datetime.date(2020, 10, 31),

22
app/model/sender_direction_statistic.py 100644
Wyświetl plik

@ -0,0 +1,22 @@
from app import db
from sqlalchemy.dialects.postgresql import JSON
class SenderDirectionStatistic(db.Model):
__tablename__ = "sender_direction_statistics"
id = db.Column(db.Integer, primary_key=True)
directions_count = db.Column(db.Integer)
messages_count = db.Column(db.Integer)
direction_data = db.Column(db.JSON)
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True)
sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("direction_stats", order_by=directions_count.desc()))
receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True)
receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("direction_stats", order_by=directions_count.desc()))
__table_args__ = (db.Index('idx_sender_direction_statistics_uc', 'sender_id', 'receiver_id', unique=True), )

19
app/model/device_info.py → app/model/sender_info.py
Wyświetl plik

@ -1,15 +1,16 @@
from app import db
from .device_info_origin import DeviceInfoOrigin
from .sender_info_origin import SenderInfoOrigin
from .aircraft_type import AircraftType
#from sqlalchemy.dialects.postgresql import ENUM
class DeviceInfo(db.Model):
__tablename__ = "device_infos"
class SenderInfo(db.Model):
__tablename__ = "sender_infos"
id = db.Column(db.Integer, primary_key=True)
address = db.Column(db.String(6), index=True)
address_type = None
# address = db.Column(db.String(6), index=True)
address = db.Column(db.String, index=True)
aircraft = db.Column(db.String)
registration = db.Column(db.String(7))
competition = db.Column(db.String(3))
@ -17,10 +18,14 @@ class DeviceInfo(db.Model):
identified = db.Column(db.Boolean)
aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN)
address_origin = db.Column(db.Enum(DeviceInfoOrigin), nullable=False, default=DeviceInfoOrigin.UNKNOWN)
address_origin = db.Column(db.Enum(SenderInfoOrigin), nullable=False, default=SenderInfoOrigin.UNKNOWN)
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id"), index=True)
sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("infos", order_by=address_origin))
def __repr__(self):
return "<DeviceInfo: %s,%s,%s,%s,%s,%s,%s,%s,%s>" % (
return "<SenderInfo: %s,%s,%s,%s,%s,%s,%s,%s,%s>" % (
self.address_type,
self.address,
self.aircraft,

9
app/model/sender_info_origin.py 100644
Wyświetl plik

@ -0,0 +1,9 @@
import enum
class SenderInfoOrigin(enum.Enum):
# lower number == more trustworthy
USER_DEFINED = 0
OGN_DDB = 1
FLARMNET = 2
UNKNOWN = 3

62
app/model/sender_position.py 100644
Wyświetl plik

@ -0,0 +1,62 @@
from geoalchemy2.types import Geometry
from app import db
from .aircraft_type import AircraftType
class SenderPosition(db.Model):
__tablename__ = "sender_positions"
reference_timestamp = db.Column(db.DateTime, primary_key=True)
# APRS data
name = db.Column(db.String)
dstcall = db.Column(db.String)
relay = db.Column(db.String)
receiver_name = db.Column(db.String(9))
timestamp = db.Column(db.DateTime)
location = db.Column("location", Geometry("POINT", srid=4326))
symboltable = None
symbolcode = None
track = db.Column(db.SmallInteger)
ground_speed = db.Column(db.Float(precision=2))
altitude = db.Column(db.Float(precision=2))
comment = None
# Type information
beacon_type = None
aprs_type = None
# Debug information
raw_message = None
# Flarm specific data
address_type = db.Column(db.SmallInteger)
aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN)
stealth = db.Column(db.Boolean)
address = db.Column(db.String)
climb_rate = db.Column(db.Float(precision=2))
turn_rate = db.Column(db.Float(precision=2))
signal_quality = db.Column(db.Float(precision=2))
error_count = db.Column(db.SmallInteger)
frequency_offset = db.Column(db.Float(precision=2))
gps_quality_horizontal = db.Column(db.SmallInteger)
gps_quality_vertical = db.Column(db.SmallInteger)
software_version = db.Column(db.Float(precision=2))
hardware_version = db.Column(db.SmallInteger)
real_address = db.Column(db.String(6))
signal_power = db.Column(db.Float(precision=2))
#proximity = None
# Calculated values (from parser)
distance = db.Column(db.Float(precision=2))
bearing = db.Column(db.SmallInteger)
normalized_quality = db.Column(db.Float(precision=2)) # signal quality normalized to 10km
# Calculated values (from this software)
location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars)
location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool
agl = db.Column(db.Float(precision=2))

24
app/model/sender_position_statistic.py 100644
Wyświetl plik

@ -0,0 +1,24 @@
from app import db
from .aircraft_type import AircraftType
from sqlalchemy.dialects.postgresql import ENUM
class SenderPositionStatistic(db.Model):
__tablename__ = "sender_position_statistics"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
dstcall = db.Column(db.String)
address_type = db.Column(db.SmallInteger)
aircraft_type = db.Column(ENUM(AircraftType, create_type=False), nullable=False, default=AircraftType.UNKNOWN)
stealth = db.Column(db.Boolean)
software_version = db.Column(db.Float(precision=2))
hardware_version = db.Column(db.SmallInteger)
messages_count = db.Column(db.Integer)
__table_args__ = (db.Index('idx_sender_position_statistics_uc', 'date', 'dstcall', 'address_type', 'aircraft_type', 'stealth', 'software_version', 'hardware_version', unique=True), )

22
app/model/sender_statistic.py 100644
Wyświetl plik

@ -0,0 +1,22 @@
from app import db
class SenderStatistic(db.Model):
__tablename__ = "sender_statistics"
id = db.Column(db.Integer, primary_key=True)
date = db.Column(db.Date)
is_trustworthy = db.Column(db.Boolean)
max_distance = db.Column(db.Float(precision=2))
max_normalized_quality = db.Column(db.Float(precision=2))
messages_count = db.Column(db.Integer)
coverages_count = db.Column(db.Integer)
receivers_count = db.Column(db.Integer)
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True)
sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("statistics", order_by=date.desc()))
__table_args__ = (db.Index('idx_sender_statistics_uc', 'date', 'sender_id', 'is_trustworthy', unique=True), )

15
app/model/takeoff_landing.py
Wyświetl plik

@ -4,13 +4,20 @@ from app import db
class TakeoffLanding(db.Model):
__tablename__ = "takeoff_landings"
device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), primary_key=True)
airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="SET NULL"), primary_key=True)
timestamp = db.Column(db.DateTime, primary_key=True)
id = db.Column(db.Integer, primary_key=True)
timestamp = db.Column(db.DateTime)
is_takeoff = db.Column(db.Boolean)
track = db.Column(db.SmallInteger)
# Relations
sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"))
sender = db.relationship("Sender", foreign_keys=[sender_id], backref="takeoff_landings")
airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="SET NULL"))
airport = db.relationship("Airport", foreign_keys=[airport_id], backref="takeoff_landings")
device = db.relationship("Device", foreign_keys=[device_id], backref="takeoff_landings", order_by="TakeoffLanding.timestamp")
country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="CASCADE"), index=True)
country = db.relationship("Country", foreign_keys=[country_id], backref="takeoff_landings")
__table_args__ = (db.Index('idx_takeoff_landings_uc', 'timestamp', 'sender_id', 'airport_id', unique=True), )

5
app/tasks/__init__.py 100644
Wyświetl plik

@ -0,0 +1,5 @@
from .sql_tasks import update_statistics, update_sender_direction_statistics
from .orm_tasks import transfer_to_database
from .orm_tasks import update_takeoff_landings, update_logbook, update_logbook_max_altitude
from .orm_tasks import import_ddb

52
app/tasks/orm_tasks.py 100644
Wyświetl plik

@ -0,0 +1,52 @@
from datetime import datetime, timedelta
from app.collect.logbook import update_takeoff_landings as logbook_update_takeoff_landings, update_logbook as logbook_update
from app.collect.logbook import update_max_altitudes as logbook_update_max_altitudes
from app.collect.database import import_ddb as device_infos_import_ddb
from app.collect.gateway import transfer_from_redis_to_database
from app import db, celery
@celery.task(name="transfer_to_database")
def transfer_to_database():
"""Transfer APRS data from Redis to database."""
result = transfer_from_redis_to_database()
return result
@celery.task(name="update_takeoff_landings")
def update_takeoff_landings(last_minutes):
"""Compute takeoffs and landings."""
end = datetime.utcnow()
start = end - timedelta(minutes=last_minutes)
result = logbook_update_takeoff_landings(start=start, end=end)
return result
@celery.task(name="update_logbook")
def update_logbook(offset_days=None):
"""Add/update logbook entries."""
result = logbook_update(offset_days=offset_days)
return result
@celery.task(name="update_logbook_max_altitude")
def update_logbook_max_altitude():
"""Add max altitudes in logbook when flight is complete (takeoff and landing)."""
result = logbook_update_max_altitudes()
return result
@celery.task(name="import_ddb")
def import_ddb():
"""Import registered devices from the DDB."""
result = device_infos_import_ddb()
return result

126
app/tasks/sql_tasks.py 100644
Wyświetl plik

@ -0,0 +1,126 @@
from datetime import datetime, timedelta
from app import db, celery
@celery.task(name="update_statistics")
def update_statistics(date_str=None):
""" Update relation_statistics, sender_statistics, receiver_statistics (all depend on coverage_statistics)."""
if date_str is None:
date_str = datetime.utcnow().strftime("%Y-%m-%d")
# Update relation statistics
db.session.execute(f"""
DELETE FROM relation_statistics
WHERE date = '{date_str}';
INSERT INTO relation_statistics AS rs (date, sender_id, receiver_id, is_trustworthy, max_distance, max_normalized_quality, messages_count, coverages_count)
SELECT
tmp.date,
tmp.sender_id,
tmp.receiver_id,
is_trustworthy,
MAX(tmp.max_distance) AS max_distance,
MAX(tmp.max_normalized_quality) AS max_normalized_quality,
SUM(tmp.messages_count) AS messages_count,
COUNT(DISTINCT tmp.location_mgrs_short) AS coverages_count
FROM coverage_statistics AS tmp
WHERE tmp.date = '{date_str}'
GROUP BY date, sender_id, receiver_id, is_trustworthy;
""")
# Update sender statistics
db.session.execute(f"""
DELETE FROM sender_statistics
WHERE date = '{date_str}';
INSERT INTO sender_statistics AS rs (date, sender_id, is_trustworthy, max_distance, max_normalized_quality, messages_count, coverages_count, receivers_count)
SELECT
tmp.date,
tmp.sender_id,
is_trustworthy,
MAX(tmp.max_distance) AS max_distance,
MAX(tmp.max_normalized_quality) AS max_normalized_quality,
SUM(tmp.messages_count) AS messages_count,
COUNT(DISTINCT tmp.location_mgrs_short) AS coverages_count,
COUNT(DISTINCT tmp.receiver_id) AS receivers_count
FROM coverage_statistics AS tmp
WHERE tmp.date = '{date_str}'
GROUP BY date, sender_id, is_trustworthy;
""")
# Update receiver statistics
db.session.execute(f"""
DELETE FROM receiver_statistics
WHERE date = '{date_str}';
INSERT INTO receiver_statistics AS rs (date, receiver_id, is_trustworthy, max_distance, max_normalized_quality, messages_count, coverages_count, senders_count)
SELECT
tmp.date,
tmp.receiver_id,
is_trustworthy,
MAX(tmp.max_distance) AS max_distance,
MAX(tmp.max_normalized_quality) AS max_normalized_quality,
SUM(tmp.messages_count) AS messages_count,
COUNT(DISTINCT tmp.location_mgrs_short) AS coverages_count,
COUNT(DISTINCT tmp.sender_id) AS senders_count
FROM coverage_statistics AS tmp
WHERE tmp.date = '{date_str}'
GROUP BY date, receiver_id, is_trustworthy;
""")
db.session.commit()
@celery.task(name="update_sender_direction_statistics")
def update_sender_direction_statistics():
""" Update sender_direction_statistics."""
db.session.execute("""
DELETE FROM sender_direction_statistics;
INSERT INTO sender_direction_statistics(sender_id, receiver_id, directions_count, messages_count, direction_data)
SELECT
sq2.sender_id,
sq2.receiver_id,
COUNT(sq2.*) AS directions_count,
SUM(sq2.messages_count) AS messages_count,
json_agg(json_build_object('direction', direction, 'messages_count', messages_count, 'max_range', max_range)) AS direction_data
FROM (
SELECT
sq.sender_id,
sq.receiver_id,
sq.direction,
COUNT(sq.*) AS messages_count,
MAX(sq.max_range) AS max_range
FROM (
SELECT
s.id AS sender_id,
r.id AS receiver_id,
10000 * 10^(sp.normalized_quality/20.0) AS max_range,
CASE
WHEN sp.bearing-sp.track < 0
THEN CAST((sp.bearing-sp.track+360)/10 AS INTEGER)*10
ELSE CAST((sp.bearing-sp.track)/10 AS INTEGER)*10
END AS direction
FROM sender_positions AS sp
INNER JOIN senders s ON sp.name = s.name
INNER JOIN receivers r ON sp.receiver_name = r.name
WHERE
sp.track IS NOT NULL AND sp.bearing IS NOT NULL AND sp.normalized_quality IS NOT NULL
AND sp.agl >= 200
AND turn_rate BETWEEN -10.0 AND 10.0
AND climb_rate BETWEEN -3.0 AND 3.0
) AS sq
GROUP BY sq.sender_id, sq.receiver_id, sq.direction
ORDER BY sq.sender_id, sq.receiver_id, sq.direction
) AS sq2
GROUP BY sq2.sender_id, sq2.receiver_id;
""")

35
app/templates/airport_detail.html
Wyświetl plik

@ -21,21 +21,36 @@
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Seen Devices</h3></div>
<div class="panel-heading"><h3 class="panel-title">Receivers</h3></div>
<table class="datatable table table-striped table-bordered">
<tr>
<th>Name</th>
</tr>
{% for receiver in airport.receivers %}
<tr>
<td><a href="{{ url_for('main.receiver_detail', receiver_id=receiver.id) }}">{{ receiver.name }}</a></td>
</tr>
{% endfor %}
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Seen Senders</h3></div>
<table class="datatable table table-striped table-bordered">
<tr>
<th>Address</th>
<th>Registration</th>
<th>Last takeoff/landing</th>
<th>Software version</th>
<th>Name</th>
<th>Last takeoff/landing</th>
<th>Hardware version</th>
<th>Software version</th>
</tr>
{% for device in devices %}
{% for sender in senders %}
<tr>
<td><a href="{{ url_for('main.device_detail', id=device.id) }}">{{ device.address }}</a></td>
<td>{% if device.info is none %}-{% else %}{{ device.info.registration }}{% endif %}</a></td>
<td>{% if device.takeoff_landings %}{% set last_action = device.takeoff_landings|last %}{% if last_action.is_takeoff == True %}↗{% else %}↘{% endif %} @ {{ last_action.timestamp.strftime('%Y-%m-%d %H:%M:%S') }}{% endif %}
<td>{% if device.software_version is not none %}{{ device.software_version }}{% else %}-{% endif %}</td>
<td>{{ sender|to_html_link|safe }}</td>
<td>{% if sender.takeoff_landings %}{% set last_action = sender.takeoff_landings|last %}{% if last_action.is_takeoff == True %}↗{% else %}↘{% endif %} @ {{ last_action.timestamp.strftime('%Y-%m-%d %H:%M:%S') }}{% endif %}
<td>{% if sender.hardware_version is not none %}{{ sender.hardware_version }}{% else %}-{% endif %}</td>
<td>{% if sender.software_version is not none %}{{ sender.software_version }}{% else %}-{% endif %}</td>
</tr>
{% endfor %}
</table>

6
app/templates/airports.html
Wyświetl plik

@ -23,6 +23,7 @@
<table class="datatable table table-striped table-bordered">
<tr>
<th>#</th>
<th>Country</th>
<th>Name</th>
<th>Logbook (takeoff and landings)</th>
</tr>
@ -30,8 +31,9 @@
{% for airport in airports %}
<tr>
<td>{{ loop.index }}
<td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ sel_country|lower }}" alt="{{ sel_country }}"/> <a href="{{ url_for('main.airport_detail', airport=airport.id) }}">{{ airport.name }}</a></td>
<td><a href="{{ url_for('main.logbook', country=sel_country, airport=airport.id) }}">Logbook</a></td>
<td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ sel_country|lower }}" alt="{{ sel_country }}"/></td>
<td>{{ airport|to_html_link|safe }}</td>
<td><a href="{{ url_for('main.logbooks', country=sel_country, airport_id=airport.id) }}">Logbook</a></td>
</tr>
{% endfor %}
</table>

11
app/templates/base.html
Wyświetl plik

@ -5,7 +5,7 @@
{% endblock %}
{% block navbar %}
<link href="http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css">
<nav class="navbar navbar-default">
<div class="container">
<div class="navbar-header">
@ -15,16 +15,17 @@
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="#">OGN DDB</a>
<a class="navbar-brand" href="#">OGN</a>
</div>
<div class="collapse navbar-collapse" id="bs-navbar-collapse-1">
<ul class="nav navbar-nav">
<li><a href="{{ url_for('main.index') }}">Home</a></li>
<li><a href="{{ url_for('main.devices') }}">Devices</a></li>
<li><a href="{{ url_for('main.senders') }}">Senders</a></li>
<li><a href="{{ url_for('main.receivers') }}">Receivers</a></li>
<li><a href="{{ url_for('main.airports') }}">Airports</a></li>
<li><a href="{{ url_for('main.logbook') }}">Logbook</a></li>
<li><a href="{{ url_for('main.statistics') }}">Statistics</a></li>
<li><a href="{{ url_for('main.logbooks') }}">Logbook</a></li>
<li><a href="{{ url_for('main.sender_ranking') }}">Sender Ranking</a></li>
<li><a href="{{ url_for('main.receiver_ranking') }}">Receiver Ranking</a></li>
</ul>
</div>
</div>

87
app/templates/device_detail.html
Wyświetl plik

@ -1,87 +0,0 @@
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Device Details</h3></div>
<table class="datatable table table-striped table-bordered">
<tr><td>Name:</td><td>{{ device.name }}</td></tr>
<tr><td>Address:</td><td>{{ device.address }}</td></tr>
<tr><td>Real Address:</td><td>{{ device.real_address }}</td></tr>
<tr><td>Stealth:</td><td>{{ device.stealth }}</td></tr>
<tr><td>Aircraft Type:</td><td>{{ device.aircraft_type }}</td></tr>
<tr><td>Software Version:</td><td>{{ device.software_version }}</td></tr>
<tr><td>Hardware Version:</td><td>{{ device.hardware_version }}</td></tr>
<tr><td>First seen:</td><td>{{ device.firstseen }}</td></tr>
<tr><td>Last seen:</td><td>{{ device.lastseen }}</td></tr>
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Device Info</h3></div>
<table class="datatable table table-striped table-bordered">
<tr>
<th>Aircraft</th>
<th>Registration</th>
<th>Competition Sign</th>
<th>Aircraft Type</th>
<th>Source</th>
</tr>
{% for info in device.get_infos() %}
<tr>
<td>{{ info.aircraft }}</td>
<td>{{ info.registration }}</td>
<td>{{ info.competition }}</td>
<td>{{ info.aircraft_type }}</td>
<td>{{ info.address_origin }}</td>
</tr>
{% endfor %}
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Logbook</h3></div>
<table class="datatable table table-striped table-bordered">
<theader>
<tr>
<th></th>
<th></th>
<th colspan="2">Airport</th>
<th colspan="2">Time UTC</th>
<th></th>
<th></th>
</tr>
<tr>
<th>#</th>
<th>Date</th>
<th>Takeoff</th>
<th>Landing</th>
<th>Takeoff</th>
<th>Landing</th>
<th>Duration</th>
<th>AGL</th>
</tr>
</theader>
<tbody>
{% set ns = namespace(mydate=none) %}
{% for entry in device.logbook %}
<tr>
<td>{{ loop.index }}</td>
<td>{% if ns.mydate != entry.reftime.strftime('%Y-%m-%d') %}{% set ns.mydate = entry.reftime.strftime('%Y-%m-%d') %}{{ ns.mydate }}{% endif %}</td>
<td>{% if entry.takeoff_airport is not none %}<a href="{{ url_for('main.airport_detail', airport=entry.takeoff_airport.id) }}">{{ entry.takeoff_airport.name }}</a>{% endif %}</td>
<td>{% if entry.landing_airport is not none %}<a href="{{ url_for('main.airport_detail', airport=entry.landing_airport.id) }}">{{ entry.landing_airport.name }}</a>{% endif %}</td>
<td>{% if entry.takeoff_timestamp is not none %} {{ entry.takeoff_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.landing_timestamp is not none %} {{ entry.landing_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.duration is not none %}{{ entry.duration }}{% endif %}</td>
<td>{% if entry.max_altitude is not none %}{{ '%0.1f'|format(entry.max_altitude - entry.takeoff_airport.altitude) }} m{% endif %}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock %}

27
app/templates/devices.html
Wyświetl plik

@ -1,27 +0,0 @@
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Devices</h3></div>
<div class="panel-body">
<table class="datatable table table-striped table-bordered">
<tr>
<th>Address</th>
<th>Registration</th>
<th>Software version</th>
</tr>
{% for device in devices %}
<tr>
<td><a href="{{ url_for('main.device_detail', id=device.id) }}">{{ device.address }}</a></td>
<td>{% if device.info is not none %}{{ device.info.registration }}{% else %} - {% endif %}</td>
<td {% if device.software_version and device.software_version < 6.6 %}class="danger"{% endif %}>{% if device.software_version is not none %}{{ device.software_version }}{% else %} - {% endif %}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}

78
app/templates/index.html 100644
Wyświetl plik

@ -0,0 +1,78 @@
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Today</h3></div>
<table class="datatable table table-striped table-bordered">
<theader>
<tr>
<th class="text-right">Senders</th>
<th class="text-right">Receivers</th>
<th class="text-right">Takeoffs</th>
<th class="text-right">Landings</th>
<th class="text-right">Sender Positions</th>
<th class="text-right">Sender Positions Total</th>
</tr>
</theader>
<tbody>
<tr>
<td class="text-right">{{ senders_today }}</td>
<td class="text-right">{{ receivers_today }}</td>
<td class="text-right">{{ takeoffs_today }}</td>
<td class="text-right">{{ landings_today }}</td>
<td class="text-right">{{ sender_positions_today }}</td>
<td class="text-right">{{ sender_positions_total }}</td>
</tr>
</tbody>
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Logbook</h3></div>
<table class="datatable table table-striped table-bordered">
<theader>
<tr>
<th></th>
<th></th>
<th colspan="2">Airport</th>
<th colspan="2">Time UTC</th>
<th></th>
<th></th>
</tr>
<tr>
<th>#</th>
<th>Date</th>
<th>Takeoff</th>
<th>Landing</th>
<th>Takeoff</th>
<th>Landing</th>
<th>Duration</th>
<th>AGL</th>
</tr>
</theader>
<tbody>
{% set ns = namespace(mydate=none) %}
{% for entry in logbook %}
<tr>
<td>{{ loop.index }}</td>
<td>{% if ns.mydate != entry.reference_timestamp.strftime('%Y-%m-%d') %}{% set ns.mydate = entry.reference_timestamp.strftime('%Y-%m-%d') %}{{ ns.mydate }}{% endif %}</td>
<td>{% if entry.takeoff_airport is not none %}<a href="{{ url_for('main.logbooks', country=entry.takeoff_airport.country_code, airport_id=entry.takeoff_airport.id, date=entry.reference_timestamp.strftime('%Y-%m-%d')) }}">{{ entry.takeoff_airport.name }}</a>{% endif %}</td>
<td>{% if entry.landing_airport is not none %}<a href="{{ url_for('main.logbooks', country=entry.landing_airport.country_code, airport_id=entry.landing_airport.id, date=entry.reference_timestamp.strftime('%Y-%m-%d')) }}">{{ entry.landing_airport.name }}</a>{% endif %}</td>
<td>{% if entry.takeoff_timestamp is not none %} {{ entry.takeoff_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.landing_timestamp is not none %} {{ entry.landing_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.duration is not none %}{{ entry.duration }}{% endif %}</td>
<td>{% if entry.max_altitude is not none %}{{ '%0.1f'|format(entry.max_altitude - entry.takeoff_airport.altitude) }} m{% endif %}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock %}

75
app/templates/logbook.html
Wyświetl plik

@ -1,75 +0,0 @@
{% extends "base.html" %}
{% block content %}
<link rel="stylesheet" href="{{ url_for('static', filename='css/flags/flags.css') }}"/>
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Logbook</h3></div>
<div class="panel-body">
<form>
<div class="well">
<select name="country" onchange="this.form.submit();">
<option value="">(none)</option>
{% for country in countries %}
<option value="{{ country.iso2 }}"{% if sel_country == country.iso2 %} SELECTED{% endif %}>{{ country.iso2 }}</option>
{% endfor %}
</select>
<select name="airport" onchange="this.form.submit();">
<option value="">(none)</option>
{% for airport in airports %}
<option value="{{ airport.id }}"{% if sel_airport == airport.id %} SELECTED{% endif %}>{{ airport.name }}</option>
{% endfor %}
</select>
<select name="date" onchange="this.form.submit();">
{% if dates|length %}
{% for entry in dates %}
<option value="{{ entry.date }}"{% if sel_date|string() == entry.date|string() %} SELECTED{% endif %}>{{ entry.date }} ({{ entry.logbook_count }})</option>
{% endfor %}
{% else %}
<option value="">(none)</option>
{% endif %}
</select>
</div>
</form>
{% if logbook is not none %}
<table class="datatable table table-striped table-bordered">
<tr>
<th>Nr.</th>
<th>Aircraft</th>
<th>Type</th>
<th colspan="2">Takeoff</th>
<th colspan="2">Landing</th>
<th>Time</th>
<th>AGL</th>
<th>Remark</th>
</tr>
{% for entry in logbook %}
<tr>
<td>{{ loop.index }}</td>
<td><a href="{{ url_for('main.device_detail', id=entry.device.id) }}">{% if entry.device.info is not none and entry.device.info.registration|length %}{{ entry.device.info.registration }}{% else %}[{{ entry.device.address }}]{% endif %}</a></td>
<td>{% if entry.device.info is not none and entry.device.info.aircraft|length %}{{ entry.device.info.aircraft }}{% else %}-{% endif %}</td>
<td>{% if entry.takeoff_timestamp is not none and entry.takeoff_airport.id == sel_airport %} {{ entry.takeoff_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.takeoff_track is not none and entry.takeoff_airport.id == sel_airport %} {{ '%02d' | format(entry.takeoff_track/10) }} {% endif %}</td>
<td>{% if entry.landing_timestamp is not none and entry.landing_airport.id == sel_airport %} {{ entry.landing_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.landing_track is not none and entry.landing_airport.id == sel_airport %} {{ '%02d' | format(entry.landing_track/10) }} {% endif %}</td>
<td>{% if entry.duration is not none %}{{ entry.duration }}{% endif %}</td>
<td>{% if entry.max_altitude is not none %}{{ '%0.1f'|format(entry.max_altitude - entry.takeoff_airport.altitude) }} m{% endif %}</td>
<td>
{% if entry.takeoff_airport is not none and entry.takeoff_airport.id != sel_airport %}Take Off: <img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ entry.takeoff_airport.country_code|lower }}" alt="{{ entry.takeoff_airport.country_code }}"/> <a href="{{ url_for('main.logbook', country=entry.takeoff_airport.country_code, airport=entry.takeoff_airport.id, date=sel_date) }}">{{ entry.takeoff_airport.name }}</a>
{% elif entry.landing_airport is not none and entry.landing_airport.id != sel_airport %}Landing: <img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ entry.landing_airport.country_code|lower }}" alt="{{ entry.landing_airport.country_code }}"/> <a href="{{ url_for('main.logbook', country=entry.takeoff_airport.country_code, airport=entry.landing_airport.id, date=sel_date) }}">{{ entry.landing_airport.name }}</a>
{% endif %}
</td>
</tr>
{% endfor %}
</table>
{% endif %}
</div>
</div>
{% endblock %}

75
app/templates/logbooks.html 100644
Wyświetl plik

@ -0,0 +1,75 @@
{% extends "base.html" %}
{% block content %}
<link rel="stylesheet" href="{{ url_for('static', filename='css/flags/flags.css') }}"/>
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Logbook</h3></div>
<div class="panel-body">
<form>
<div class="well">
<select name="country" onchange="this.form.submit();">
<option value="">(none)</option>
{% for country in countries %}
<option value="{{ country.iso2 }}"{% if sel_country == country.iso2 %} SELECTED{% endif %}>{{ country.iso2 }}</option>
{% endfor %}
</select>
<select name="airport_id" onchange="this.form.submit();">
<option value="">(none)</option>
{% for airport in airports %}
<option value="{{ airport.id }}"{% if sel_airport_id == airport.id %} SELECTED{% endif %}>{{ airport.name }}</option>
{% endfor %}
</select>
<select name="date" onchange="this.form.submit();">
{% if dates|length %}
{% for entry in dates %}
<option value="{{ entry.date }}"{% if sel_date|string() == entry.date|string() %} SELECTED{% endif %}>{{ entry.date }} ({{ entry.logbook_count }})</option>
{% endfor %}
{% else %}
<option value="">(none)</option>
{% endif %}
</select>
</div>
</form>
{% if logbooks is not none %}
<table class="datatable table table-striped table-bordered">
<tr>
<th>#</th>
<th>Aircraft</th>
<th>Type</th>
<th colspan="2">Takeoff</th>
<th colspan="2">Landing</th>
<th>Time</th>
<th>AGL</th>
<th>Remark</th>
</tr>
{% for entry in logbooks %}
<tr>
<td>{{ loop.index }}</td>{% set sender = entry.sender %}
<td>{{ sender|to_html_link|safe }}</td>
<td>{% if sender.infos|length > 0 and sender.infos[0].aircraft|length %}{{ sender.infos[0].aircraft }}{% else %}-{% endif %}</td>
<td>{% if entry.takeoff_timestamp is not none and entry.takeoff_airport.id == sel_airport_id %} {{ entry.takeoff_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.takeoff_track is not none and entry.takeoff_airport.id == sel_airport_id %} {{ '%02d' | format(entry.takeoff_track/10) }} {% endif %}</td>
<td>{% if entry.landing_timestamp is not none and entry.landing_airport.id == sel_airport_id %} {{ entry.landing_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.landing_track is not none and entry.landing_airport.id == sel_airport_id %} {{ '%02d' | format(entry.landing_track/10) }} {% endif %}</td>
<td>{% if entry.duration is not none %}{{ entry.duration }}{% endif %}</td>
<td>{% if entry.max_altitude is not none %}{{ '%d' | format(entry.max_altitude - entry.takeoff_airport.altitude) }} m{% endif %}</td>
<td>
{% if entry.takeoff_airport is not none and entry.takeoff_airport.id != sel_airport_id %}Take Off: <img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ entry.takeoff_airport.country_code|lower }}" alt="{{ entry.takeoff_airport.country_code }}"/> <a href="{{ url_for('main.logbooks', country=entry.takeoff_airport.country_code, airport_id=entry.takeoff_airport.id, date=sel_date) }}">{{ entry.takeoff_airport.name }}</a>
{% elif entry.landing_airport is not none and entry.landing_airport.id != sel_airport_id %}Landing: <img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ entry.landing_airport.country_code|lower }}" alt="{{ entry.landing_airport.country_code }}"/> <a href="{{ url_for('main.logbooks', country=entry.takeoff_airport.country_code, airport_id=entry.landing_airport.id, date=sel_date) }}">{{ entry.landing_airport.name }}</a>
{% endif %}
</td>
</tr>
{% endfor %}
</table>
{% endif %}
</div>
</div>
{% endblock %}

28
app/templates/receiver_detail.html
Wyświetl plik

@ -12,18 +12,38 @@
<table class="datatable table table-striped table-bordered">
<tr><td>Name:</td><td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ receiver.country.iso2|lower }}" alt="{{ receiver.country.iso2 }}"/> {{ receiver.name }}</td></tr>
<tr><td>Airport:</td>
<td>{% if airport is not none %}<img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ airport.country_code|lower }}" alt="{{ airport.country_code }}"/>
{% if airport.takeoff_landings %}<a href="{{ url_for('main.airport_detail', airport=airport.id) }}">{{ airport.name }}</a>{% else %}{{ airport.name }}{% endif %}
<td>{% if receiver.airport is not none %}<img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ receiver.airport.country_code|lower }}" alt="{{ receiver.airport.country_code }}"/>
<a href="{{ url_for('main.airport_detail', airport_id=receiver.airport.id) }}">{{ receiver.airport.name }}</a>
{% else %}-{% endif %}
</td>
</tr>
<tr><td>Version:</td><td>{{ receiver.version }}</td></tr>
<tr><td>Platform:</td><td>{{ receiver.platform }}</td></tr>
<tr><td>Altitude:</td><td>{{ receiver.altitude|int }}m</td></tr>
<tr><td>AGL:</td><td>{{ receiver.agl|int }}m</td></tr>
<tr><td>Version:</td><td>{{ receiver.version if receiver.version else '-' }}</td></tr>
<tr><td>Platform:</td><td>{{ receiver.platform if receiver.platform else '-' }}</td></tr>
<tr><td>First seen:</td><td>{{ receiver.firstseen }}</td></tr>
<tr><td>Last seen:</td><td>{{ receiver.lastseen }}</td></tr>
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Airport nearby</h3></div>
<table class="datatable table table-striped table-bordered">
<tr>
<th>#</th>
<th>Name</th>
<th class="text-right">Distance [km]</th>
</tr>
{% for (airport,distance,azimuth) in receiver.airports_nearby() %}
<tr>
<td>{{ loop.index }}</td>
<td>{% if airport.takeoff_landings|length > 0 %}{{ airport|to_html_link|safe }}{% else %}{{ airport.name }}{% endif %}</td>
<td class="text-right">{{ '%0.1f' | format(distance/1000.0) }} ({{ azimuth|to_ordinal }})</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}

41
app/templates/receiver_ranking.html 100644
Wyświetl plik

@ -0,0 +1,41 @@
{% extends "base.html" %}
{% block content %}
<link rel="stylesheet" href="{{ url_for('static', filename='css/flags/flags.css') }}"/>
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Receiver Ranking</h3></div>
<div class="panel-body">
<table id="myTable" class="table table-striped table-bordered tablesorter tablesorter-bootstrap">
<tr>
<th>#</th>
<th>Country</th>
<th>Name</th>
<th>Airport</th>
<th class="text-right">Maximum distance [km]</th>
<th class="text-right">Maximal normalized signal quality [dB]</th>
<th class="text-right">Sender counter</th>
<th class="text-right">Coverage counter</th>
<th class="text-right">Message counter</th>
</tr>
{% for entry in ranking %}
<tr>
<td>{{ loop.index }}</td>
<td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ entry.receiver.country.iso2|lower }}" alt="{{ entry.receiver.country.iso2 }}"/></td>
<td>{{ entry.receiver|to_html_link|safe }}</a></td>
<td>{{ entry.receiver.airport|to_html_link|safe }}</a></td>
<td class="text-right">{{ '%0.1f' | format(entry.max_distance/1000.0) }}</td>
<td class="text-right">{{ '%0.1f' | format(entry.max_normalized_quality) }}</td>
<td class="text-right">{{ entry.senders_count }}</td>
<td class="text-right">{{ entry.coverages_count }}</td>
<td class="text-right">{{ entry.messages_count }}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}

18
app/templates/receivers.html
Wyświetl plik

@ -22,14 +22,26 @@
<table class="datatable table table-striped table-bordered">
<tr>
<th>Name</th>
<th>#</th>
<th>Country</th>
<th>Name</th>
<th>Airport</th>
<th>Altitude</th>
<th>Status</th>
<th>Version</th>
<th>Platform</th>
</tr>
{% for receiver in receivers %}
<tr>
<td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ receiver.country.iso2|lower }}" alt="{{ receiver.country.iso2 }}"/> <a href="{{ url_for('main.receiver_detail', receiver_id=receiver.id) }}">{{ receiver.name }}</a></td>
<td>{{ receiver.altitude|int }} m</td>
<td>{{ loop.index }}</td>
<td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ receiver.country.iso2|lower }}" alt="{{ receiver.country.iso2 }}"/></td>
<td>{{ receiver|to_html_link|safe }}</td>
<td>{{ receiver.airport|to_html_link|safe }}</td>
<td>{{ receiver.altitude|int }} m</td>
<td>{{ receiver.lastseen|timestamp_to_status|safe }}</td>
<td>{{ receiver.version if receiver.version else '-' }}</td>
<td>{{ receiver.platform if receiver.platform else '-' }}</td>
</tr>
{% endfor %}
</table>

92
app/templates/sender_detail.html 100644
Wyświetl plik

@ -0,0 +1,92 @@
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Sender Details</h3></div>
<table class="datatable table table-striped table-bordered">
<tr><td>Name:</td><td>{{ sender.name }}</td></tr>
<tr><td>Address:</td><td>{{ sender.address if sender.address else '-' }}</td></tr>
<tr><td>Real Address:</td><td>{{ sender.real_address if sender.real_address else '-' }}</td></tr>
<tr><td>Stealth:</td><td>{{ sender.stealth if sender.stealth else '-' }}</td></tr>
<tr><td>Aircraft Type:</td><td>{{ sender.aircraft_type.name }}</td></tr>
<tr><td>Software Version:</td><td>{{ sender.software_version if sender.software_version else '-' }}</td></tr>
<tr><td>Hardware Version:</td><td>{{ sender.hardware_version if sender.hardware_version else '-' }}</td></tr>
<tr><td>First seen:</td><td>{{ sender.firstseen }}</td></tr>
<tr><td>Last seen:</td><td>{{ sender.lastseen }}</td></tr>
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Sender Info</h3></div>
<table class="datatable table table-striped table-bordered">
<tr>
<th>Aircraft</th>
<th>Registration</th>
<th>Competition Sign</th>
<th>Aircraft Type</th>
<th>Source</th>
</tr>
{% for info in sender.infos %}
<tr>
<td>{{ info.aircraft }}</td>
<td>{{ info.registration }}</td>
<td>{{ info.competition }}</td>
<td>{{ info.aircraft_type.name }}</td>
<td>{{ info.address_origin.name }}</td>
</tr>
{% endfor %}
</table>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Range View</h3></div>
<img src="{{ url_for('main.range_view', sender_id=sender.id) }}" class="img-thumbnail">
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Logbook</h3></div>
<table class="datatable table table-striped table-bordered">
<theader>
<tr>
<th></th>
<th></th>
<th colspan="2">Airport</th>
<th colspan="2">Time UTC</th>
<th></th>
<th></th>
</tr>
<tr>
<th>#</th>
<th>Date</th>
<th>Takeoff</th>
<th>Landing</th>
<th>Takeoff</th>
<th>Landing</th>
<th>Duration</th>
<th>AGL</th>
</tr>
</theader>
<tbody>
{% set ns = namespace(mydate=none) %}
{% for entry in sender.logbook_entries %}
<tr>
<td>{{ loop.index }}</td>
<td>{% if ns.mydate != entry.reference_timestamp.strftime('%Y-%m-%d') %}{% set ns.mydate = entry.reference_timestamp.strftime('%Y-%m-%d') %}{{ ns.mydate }}{% endif %}</td>
<td>{% if entry.takeoff_airport is not none %}<a href="{{ url_for('main.airport_detail', airport_id=entry.takeoff_airport.id) }}">{{ entry.takeoff_airport.name }}</a>{% endif %}</td>
<td>{% if entry.landing_airport is not none %}<a href="{{ url_for('main.airport_detail', airport_id=entry.landing_airport.id) }}">{{ entry.landing_airport.name }}</a>{% endif %}</td>
<td>{% if entry.takeoff_timestamp is not none %} {{ entry.takeoff_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.landing_timestamp is not none %} {{ entry.landing_timestamp.strftime('%H:%M') }} {% endif %}</td>
<td>{% if entry.duration is not none %}{{ entry.duration }}{% endif %}</td>
<td>{% if entry.max_altitude is not none %}{{ '%0.1f'|format(entry.max_altitude - entry.takeoff_airport.altitude) }} m{% endif %}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock %}

39
app/templates/sender_ranking.html 100644
Wyświetl plik

@ -0,0 +1,39 @@
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Sender Ranking</h3></div>
<div class="panel-body">
<table class="datatable table table-striped table-bordered">
<tr>
<th>#</th>
<th>Name</th>
<th>Aircraft</th>
<th class="text-right">Maximum distance [km]</th>
<th class="text-right">Maximal normalized signal quality [dB]</th>
<th class="text-right">Receiver counter</th>
<th class="text-right">Coverage counter</th>
<th class="text-right">Message counter</th>
</tr>
{% for entry in ranking %}
<tr>
<td>{{ loop.index }}</td>
<td>{{ entry.sender|to_html_link|safe }}</a></td>
<td>{% if entry.sender.infos|length > 0 %}{{ entry.sender.infos[0].aircraft }}{% else %}-{% endif %}</td>
<td class="text-right">{{ '%0.1f' | format(entry.max_distance/1000.0) }}</td>
<td class="text-right">{{ '%0.1f' | format(entry.max_normalized_quality) }}</td>
<td class="text-right">{{ entry.receivers_count }}</td>
<td class="text-right">{{ entry.coverages_count }}</td>
<td class="text-right">{{ entry.messages_count }}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}

29
app/templates/senders.html 100644
Wyświetl plik

@ -0,0 +1,29 @@
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-title">Senders</h3></div>
<div class="panel-body">
<table class="datatable table table-striped table-bordered">
<tr>
<th>#</th>
<th>Name</th>
<th>Registration</th>
<th>Software version</th>
</tr>
{% for sender in senders %}
<tr>
<td>{{ loop.index }}</td>
<td><a href="{{ url_for('main.sender_detail', sender_id=sender.id) }}">{{ sender.name }}</a></td>
<td>{{ sender.infos[0].registration if sender.infos|length > 0 else '-' }}</td>
<td {% if sender.software_version and sender.software_version < 6.6 %}class="danger"{% endif %}>{% if sender.software_version is not none %}{{ sender.software_version }}{% else %} - {% endif %}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}

29
app/templates/statistics.html
Wyświetl plik

@ -1,29 +0,0 @@
{% extends "base.html" %}
{% block content %}
<link rel="stylesheet" href="{{ url_for('static', filename='css/flags/flags.css') }}"/>
<div class="container">
<div class="panel panel-success" id="asdf">
<div class="panel-heading"><h3 class="panel-title">Devices</h3></div>
<div class="panel-body">
<table class="datatable table table-striped table-bordered">
<tr>
<th>Receiver</th>
<th>Aircrafts</th>
<th>Beacons</th>
</tr>
{% for receiverstat in receiverstats %}
<tr>
<td><img src="{{ url_for('static', filename='img/Transparent.gif') }}" class="flag flag-{{ receiverstat.receiver.country.iso2|lower }}" alt="{{ receiverstat.receiver.country.iso2 }}"/> {{ receiverstat.receiver.name }}</td>
<td>{{ receiverstat.aircraft_count }}</td>
<td>{{ receiverstat.beacon_count }}</td>
</tr>
{% endfor %}
</table>
</div>
</div>
{% endblock %}

71
app/utils.py
Wyświetl plik

@ -3,11 +3,13 @@ import gzip
from io import StringIO
from datetime import datetime, timedelta
from flask import current_app
from aerofiles.seeyou import Reader
from ogn.parser.utils import FEETS_TO_METER
import requests
from .model import AircraftType, DeviceInfoOrigin, DeviceInfo, Airport, Location
from .model import AircraftType, SenderInfoOrigin, SenderInfo, Airport, Location
DDB_URL = "http://ddb.glidernet.org/download/?t=1"
@ -31,7 +33,7 @@ def date_to_timestamps(date):
return (start, end)
def get_ddb(csv_file=None, address_origin=DeviceInfoOrigin.UNKNOWN):
def get_ddb(csv_file=None, address_origin=SenderInfoOrigin.UNKNOWN):
if csv_file is None:
r = requests.get(DDB_URL)
rows = "\n".join(i for i in r.text.splitlines() if i[0] != "#")
@ -41,43 +43,43 @@ def get_ddb(csv_file=None, address_origin=DeviceInfoOrigin.UNKNOWN):
data = csv.reader(StringIO(rows), quotechar="'", quoting=csv.QUOTE_ALL)
device_infos = list()
sender_infos = list()
for row in data:
device_info = DeviceInfo()
device_info.address_type = row[0]
device_info.address = row[1]
device_info.aircraft = row[2]
device_info.registration = row[3]
device_info.competition = row[4]
device_info.tracked = row[5] == "Y"
device_info.identified = row[6] == "Y"
device_info.aircraft_type = AircraftType(int(row[7]))
device_info.address_origin = address_origin
sender_info = SenderInfo()
sender_info.address_type = row[0]
sender_info.address = row[1]
sender_info.aircraft = row[2]
sender_info.registration = row[3]
sender_info.competition = row[4]
sender_info.tracked = row[5] == "Y"
sender_info.identified = row[6] == "Y"
sender_info.aircraft_type = AircraftType(int(row[7]))
sender_info.address_origin = address_origin
device_infos.append(device_info)
sender_infos.append(sender_info)
return device_infos
return sender_infos
def get_flarmnet(fln_file=None, address_origin=DeviceInfoOrigin.FLARMNET):
def get_flarmnet(fln_file=None, address_origin=SenderInfoOrigin.FLARMNET):
if fln_file is None:
r = requests.get(FLARMNET_URL)
rows = [bytes.fromhex(line).decode("latin1") for line in r.text.split("\n") if len(line) == 172]
rows = [bytes.fromhex(line).decode("latin1") for line in r.text.split("\n") if len(line) == 173]
else:
with open(fln_file, "r") as file:
rows = [bytes.fromhex(line.strip()).decode("latin1") for line in file.readlines() if len(line) == 172]
rows = [bytes.fromhex(line.strip()).decode("latin1") for line in file.readlines() if len(line) == 173]
device_infos = list()
sender_infos = list()
for row in rows:
device_info = DeviceInfo()
device_info.address = row[0:6].strip()
device_info.aircraft = row[48:69].strip()
device_info.registration = row[69:76].strip()
device_info.competition = row[76:79].strip()
sender_info = SenderInfo()
sender_info.address = row[0:6].strip()
sender_info.aircraft = row[48:69].strip()
sender_info.registration = row[69:76].strip()
sender_info.competition = row[76:79].strip()
device_infos.append(device_info)
sender_infos.append(sender_info)
return device_infos
return sender_infos
def get_trackable(ddb):
@ -118,7 +120,7 @@ def get_airports(cupfile):
airports.append(airport)
except AttributeError as e:
print("Failed to parse line: {} {}".format(line, e))
current_app.logger.error("Failed to parse line: {} {}".format(line, e))
return airports
@ -134,3 +136,18 @@ def open_file(filename):
else:
f = open(filename, "rt", encoding="latin-1")
return f
def get_sql_trustworthy(source_table_alias):
MIN_DISTANCE = 1000
MAX_DISTANCE = 640000
MAX_NORMALIZED_QUALITY = 40 # this is enough for > 640km
MAX_ERROR_COUNT = 5
MAX_CLIMB_RATE = 50
return f"""
({source_table_alias}.distance IS NOT NULL AND {source_table_alias}.distance BETWEEN {MIN_DISTANCE} AND {MAX_DISTANCE})
AND ({source_table_alias}.normalized_quality IS NOT NULL AND {source_table_alias}.normalized_quality < {MAX_NORMALIZED_QUALITY})
AND ({source_table_alias}.error_count IS NULL OR {source_table_alias}.error_count < {MAX_ERROR_COUNT})
AND ({source_table_alias}.climb_rate IS NULL OR {source_table_alias}.climb_rate BETWEEN -{MAX_CLIMB_RATE} AND {MAX_CLIMB_RATE})
"""

6
celery_app.py 100644
Wyświetl plik

@ -0,0 +1,6 @@
#!/usr/bin/env python
from app import init_celery
app = init_celery()
app.conf.imports = app.conf.imports + ("app.tasks",)

7
celery_worker.py
Wyświetl plik

@ -1,7 +0,0 @@
#!/usr/bin/env python
import os
from app import celery, create_app
from app.collect.celery import *
app = create_app(os.getenv('FLASK_CONFIG') or 'default')
app.app_context().push()

53
config.py 100644
Wyświetl plik

@ -0,0 +1,53 @@
import os
class BaseConfig:
SECRET_KEY = "i-like-ogn"
# Flask-Cache stuff
CACHE_TYPE = "simple"
CACHE_DEFAULT_TIMEOUT = 300
# Redis stuff
REDIS_URL = "redis://localhost:6379/0"
# Celery stuff
BROKER_URL = os.environ.get("BROKER_URL", REDIS_URL)
CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND", REDIS_URL)
APRS_USER = "OGNPYTHON"
class DefaultConfig(BaseConfig):
SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI", "postgresql://postgres:postgres@localhost:5432/ogn")
SQLALCHEMY_TRACK_MODIFICATIONS = False
# Celery beat stuff
from celery.schedules import crontab
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
"transfer_to_database": {"task": "transfer_to_database", "schedule": timedelta(minutes=1)},
"update_statistics": {"task": "update_statistics", "schedule": timedelta(minutes=5)},
"update_takeoff_landings": {"task": "update_takeoff_landings", "schedule": timedelta(minutes=1), "kwargs": {"last_minutes": 20}},
"update_logbook": {"task": "update_logbook", "schedule": timedelta(minutes=1)},
"update_logbook_previous_day": {"task": "update_logbook", "schedule": crontab(hour=1, minute=0), "kwargs": {"day_offset": -1}},
"update_ddb_daily": {"task": "import_ddb", "schedule": timedelta(days=1)},
#"update_logbook_max_altitude": {"task": "update_logbook_max_altitude", "schedule": timedelta(minutes=1), "kwargs": {"offset_days": 0}},
#"purge_old_data": {"task": "purge_old_data", "schedule": timedelta(hours=1), "kwargs": {"max_hours": 48}},
}
class DevelopmentConfig(BaseConfig):
SQLALCHEMY_DATABASE_URI = "postgresql://postgres:postgres@localhost:5432/ogn_test"
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_ECHO = False
configs = {
'default': DefaultConfig,
'development': DevelopmentConfig,
'testing': DevelopmentConfig
}

37
deployment/nginx/ddb
Wyświetl plik

@ -1,37 +0,0 @@
server {
# listen on port 80 (http)
listen 80;
server_name _;
location / {
# redirect any requests to the same URL but on https
return 301 https://$host$request_uri;
}
}
server {
# listen on port 443 (https)
listen 443 ssl;
server_name _;
# location of the self-signed SSL certificate
ssl_certificate /home/ubuntu/ddb/certs/cert.pem;
ssl_certificate_key /home/ubuntu/ddb/certs/key.pem;
# write access and error logs to /var/log
access_log /var/log/ddb_access.log;
error_log /var/log/ddb_error.log;
location / {
# forward application requests to the gunicorn server
proxy_pass http://localhost:8000;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /static {
# handle static files directly, without forwarding to the application
alias /home/ubuntu/ddb/app/static;
expires 30d;
}
}

20
deployment/nginx/ogn-python 100644
Wyświetl plik

@ -0,0 +1,20 @@
server {
listen 80;
server_name api.example.com;
location / {
proxy_pass "http://localhost:5000";
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
fastcgi_read_timeout 300s;
proxy_read_timeout 300;
}
location /static {
alias /home/pi/ogn-python/app/static/;
}
error_log /var/log/nginx/api-error.log;
access_log /var/log/nginx/api-access.log;
}

2
deployment/supervisor/celerybeatd.conf
Wyświetl plik

@ -1,5 +1,5 @@
[program:celerybeat]
command=/home/pi/ogn-python/venv/bin/celery -A celery_worker.celery beat -l info
command=/home/pi/ogn-python/venv/bin/celery -A celery_app beat -l info
directory=/home/pi/ogn-python
environment=FLASK_APP=ogn_python.py

2
deployment/supervisor/celeryd.conf
Wyświetl plik

@ -1,5 +1,5 @@
[program:celery]
command=/home/pi/ogn-python/venv/bin/celery -A celery_worker.celery worker -l info
command=/home/pi/ogn-python/venv/bin/celery -A celery_app worker -l info
directory=/home/pi/ogn-python
environment=FLASK_APP=ogn_python.py

2
deployment/supervisor/flower.conf
Wyświetl plik

@ -1,6 +1,6 @@
[program:flower]
environment=OGN_CONFIG_MODULE='config/default.py'
command=/home/pi/ogn-python/venv/bin/celery flower -A app.celery --port=5555 -l info
command=/home/pi/ogn-python/venv/bin/celery flower -A celery_app --port=5555 -l info
directory=/home/pi/ogn-python
user=pi

4
deployment/supervisor/ogn-feeder.conf → deployment/supervisor/ogn-gateway.conf
Wyświetl plik

@ -4,7 +4,7 @@ directory=/home/pi/ogn-python
environment=FLASK_APP=ogn_python.py
user=pi
stderr_logfile=/var/log/supervisor/ogn-feeder.log
stdout_logfile=/var/log/supervisor/ogn-feeder.log
stderr_logfile=/var/log/supervisor/ogn-gateway.log
stdout_logfile=/var/log/supervisor/ogn-gateway.log
autostart=true
autorestart=true

76
migrations/versions/079fe885ae20_remove_non_position_fields_from_.py
Wyświetl plik

@ -1,76 +0,0 @@
"""Remove non position fields from AircraftBeacon and ReceiverBeacon
Revision ID: 079fe885ae20
Revises: 6c19cedf5fa7
Create Date: 2019-09-25 21:42:34.924732
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = '079fe885ae20'
down_revision = '6c19cedf5fa7'
branch_labels = None
depends_on = None
def upgrade():
op.drop_constraint('aircraft_beacons_receiver_id_fkey', 'aircraft_beacons', type_='foreignkey')
op.drop_constraint('aircraft_beacons_device_id_fkey', 'aircraft_beacons', type_='foreignkey')
op.drop_column('aircraft_beacons', 'device_id')
op.drop_column('aircraft_beacons', 'receiver_id')
op.drop_constraint('receiver_beacons_receiver_id_fkey', 'receiver_beacons', type_='foreignkey')
op.drop_column('receiver_beacons', 'receiver_id')
op.drop_column('receiver_beacons', 'total_ram')
op.drop_column('receiver_beacons', 'senders_visible')
op.drop_column('receiver_beacons', 'senders_messages')
op.drop_column('receiver_beacons', 'cpu_temp')
op.drop_column('receiver_beacons', 'platform')
op.drop_column('receiver_beacons', 'rec_input_noise')
op.drop_column('receiver_beacons', 'ntp_error')
op.drop_column('receiver_beacons', 'good_senders')
op.drop_column('receiver_beacons', 'senders_total')
op.drop_column('receiver_beacons', 'cpu_load')
op.drop_column('receiver_beacons', 'free_ram')
op.drop_column('receiver_beacons', 'good_and_bad_senders')
op.drop_column('receiver_beacons', 'amperage')
op.drop_column('receiver_beacons', 'voltage')
op.drop_column('receiver_beacons', 'senders_signal')
op.drop_column('receiver_beacons', 'version')
op.drop_column('receiver_beacons', 'relay')
op.drop_column('receiver_beacons', 'rt_crystal_correction')
op.drop_column('receiver_beacons', 'good_senders_signal')
def downgrade():
op.add_column('receiver_beacons', sa.Column('receiver_id', sa.INTEGER(), autoincrement=False, nullable=True))
op.create_foreign_key('receiver_beacons_receiver_id_fkey', 'receiver_beacons', 'receivers', ['receiver_id'], ['id'], ondelete='SET NULL')
op.add_column('receiver_beacons', sa.Column('good_senders_signal', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('rt_crystal_correction', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('relay', sa.VARCHAR(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('version', sa.VARCHAR(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('senders_signal', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('voltage', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('amperage', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('good_and_bad_senders', sa.INTEGER(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('free_ram', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('cpu_load', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('senders_total', sa.INTEGER(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('good_senders', sa.INTEGER(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('ntp_error', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('rec_input_noise', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('platform', sa.VARCHAR(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('cpu_temp', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('senders_messages', sa.INTEGER(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('senders_visible', sa.INTEGER(), autoincrement=False, nullable=True))
op.add_column('receiver_beacons', sa.Column('total_ram', sa.REAL(), autoincrement=False, nullable=True))
op.add_column('aircraft_beacons', sa.Column('receiver_id', sa.INTEGER(), autoincrement=False, nullable=True))
op.add_column('aircraft_beacons', sa.Column('device_id', sa.INTEGER(), autoincrement=False, nullable=True))
op.create_foreign_key('aircraft_beacons_device_id_fkey', 'aircraft_beacons', 'devices', ['device_id'], ['id'], ondelete='SET NULL')
op.create_foreign_key('aircraft_beacons_receiver_id_fkey', 'aircraft_beacons', 'receivers', ['receiver_id'], ['id'], ondelete='SET NULL')

76
migrations/versions/6c19cedf5fa7_use_enum_for_aircrafttype.py
Wyświetl plik

@ -1,76 +0,0 @@
"""Use Enum for AircraftType
Revision ID: 6c19cedf5fa7
Revises: be9a6dad551e
Create Date: 2019-09-24 18:37:40.224279
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = '6c19cedf5fa7'
down_revision = 'be9a6dad551e'
branch_labels = None
depends_on = None
aircrafttype = postgresql.ENUM('UNKNOWN', 'GLIDER_OR_MOTOR_GLIDER', 'TOW_TUG_PLANE', 'HELICOPTER_ROTORCRAFT',
'PARACHUTE', 'DROP_PLANE', 'HANG_GLIDER', 'PARA_GLIDER', 'POWERED_AIRCRAFT',
'JET_AIRCRAFT', 'FLYING_SAUCER', 'BALLOON', 'AIRSHIP', 'UNMANNED_AERIAL_VEHICLE',
'STATIC_OBJECT', name='aircrafttype')
def upgrade():
aircrafttype.create(op.get_bind())
for table in ['aircraft_beacons', 'devices']:
op.add_column(table, sa.Column('aircraft_type_enum', sa.Enum(
'UNKNOWN', 'GLIDER_OR_MOTOR_GLIDER', 'TOW_TUG_PLANE', 'HELICOPTER_ROTORCRAFT',
'PARACHUTE', 'DROP_PLANE', 'HANG_GLIDER', 'PARA_GLIDER', 'POWERED_AIRCRAFT',
'JET_AIRCRAFT', 'FLYING_SAUCER', 'BALLOON', 'AIRSHIP', 'UNMANNED_AERIAL_VEHICLE',
'STATIC_OBJECT', name='aircrafttype'), nullable=False, server_default='UNKNOWN'))
op.execute("""
UPDATE {table} SET aircraft_type_enum = 'UNKNOWN' WHERE aircraft_type = 0;
UPDATE {table} SET aircraft_type_enum = 'GLIDER_OR_MOTOR_GLIDER' WHERE aircraft_type = 1;
UPDATE {table} SET aircraft_type_enum = 'TOW_TUG_PLANE' WHERE aircraft_type = 2;
UPDATE {table} SET aircraft_type_enum = 'HELICOPTER_ROTORCRAFT' WHERE aircraft_type = 3;
UPDATE {table} SET aircraft_type_enum = 'PARACHUTE' WHERE aircraft_type = 4;
UPDATE {table} SET aircraft_type_enum = 'DROP_PLANE' WHERE aircraft_type = 5;
UPDATE {table} SET aircraft_type_enum = 'HANG_GLIDER' WHERE aircraft_type = 6;
UPDATE {table} SET aircraft_type_enum = 'PARA_GLIDER' WHERE aircraft_type = 7;
UPDATE {table} SET aircraft_type_enum = 'POWERED_AIRCRAFT' WHERE aircraft_type = 8;
UPDATE {table} SET aircraft_type_enum = 'JET_AIRCRAFT' WHERE aircraft_type = 9;
UPDATE {table} SET aircraft_type_enum = 'FLYING_SAUCER' WHERE aircraft_type = 10;
UPDATE {table} SET aircraft_type_enum = 'BALLOON' WHERE aircraft_type = 11;
UPDATE {table} SET aircraft_type_enum = 'AIRSHIP' WHERE aircraft_type = 12;
UPDATE {table} SET aircraft_type_enum = 'UNMANNED_AERIAL_VEHICLE' WHERE aircraft_type = 13;
UPDATE {table} SET aircraft_type_enum = 'STATIC_OBJECT' WHERE aircraft_type = 15;
""".format(table=table))
op.drop_column(table, 'aircraft_type')
op.alter_column(table, 'aircraft_type_enum', new_column_name='aircraft_type')
def downgrade():
for table in ['aircraft_beacons', 'devices']:
op.add_column(table, sa.Column('aircraft_type_int', sa.SmallInteger))
op.execute("""
UPDATE {table} SET aircraft_type_int = 0 WHERE aircraft_type = 'UNKNOWN';
UPDATE {table} SET aircraft_type_int = 1 WHERE aircraft_type = 'GLIDER_OR_MOTOR_GLIDER';
UPDATE {table} SET aircraft_type_int = 2 WHERE aircraft_type = 'TOW_TUG_PLANE';
UPDATE {table} SET aircraft_type_int = 3 WHERE aircraft_type = 'HELICOPTER_ROTORCRAFT';
UPDATE {table} SET aircraft_type_int = 4 WHERE aircraft_type = 'PARACHUTE';
UPDATE {table} SET aircraft_type_int = 5 WHERE aircraft_type = 'DROP_PLANE';
UPDATE {table} SET aircraft_type_int = 6 WHERE aircraft_type = 'HANG_GLIDER';
UPDATE {table} SET aircraft_type_int = 7 WHERE aircraft_type = 'PARA_GLIDER';
UPDATE {table} SET aircraft_type_int = 8 WHERE aircraft_type = 'POWERED_AIRCRAFT';
UPDATE {table} SET aircraft_type_int = 9 WHERE aircraft_type = 'JET_AIRCRAFT';
UPDATE {table} SET aircraft_type_int = 10 WHERE aircraft_type = 'FLYING_SAUCER';
UPDATE {table} SET aircraft_type_int = 11 WHERE aircraft_type = 'BALLOON';
UPDATE {table} SET aircraft_type_int = 12 WHERE aircraft_type = 'AIRSHIP';
UPDATE {table} SET aircraft_type_int = 13 WHERE aircraft_type = 'UNMANNED_AERIAL_VEHICLE';
UPDATE {table} SET aircraft_type_int = 15 WHERE aircraft_type = 'STATIC_OBJECT';
""".format(table=table))
op.drop_column(table, 'aircraft_type')
op.alter_column(table, 'aircraft_type_int', new_column_name='aircraft_type')
aircrafttype.drop(op.get_bind())

28
migrations/versions/885123e6a2d6_remove_fk_relation_from_device_to_.py
Wyświetl plik

@ -1,28 +0,0 @@
"""Remove FK relation from device to device_info
Revision ID: 885123e6a2d6
Revises: 002656878233
Create Date: 2019-04-27 14:22:30.841969
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = '885123e6a2d6'
down_revision = '002656878233'
branch_labels = None
depends_on = None
def upgrade():
op.drop_index('ix_device_infos_device_id', table_name='device_infos')
op.drop_constraint('device_infos_device_id_fkey', 'device_infos', type_='foreignkey')
op.drop_column('device_infos', 'device_id')
def downgrade():
op.add_column('device_infos', sa.Column('device_id', sa.INTEGER(), autoincrement=False, nullable=True))
op.create_foreign_key('device_infos_device_id_fkey', 'device_infos', 'devices', ['device_id'], ['id'], ondelete='SET NULL')
op.create_index('ix_device_infos_device_id', 'device_infos', ['device_id'], unique=False)

41
migrations/versions/be9a6dad551e_use_enums_for_deviceinfoorigin.py
Wyświetl plik

@ -1,41 +0,0 @@
"""Use Enums for DeviceInfoOrigin
Revision ID: be9a6dad551e
Revises: 885123e6a2d6
Create Date: 2019-09-15 14:38:25.838089
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = 'be9a6dad551e'
down_revision = '885123e6a2d6'
branch_labels = None
depends_on = None
deviceinfoorigin = postgresql.ENUM('UNKNOWN', 'OGN_DDB', 'FLARMNET', 'USER_DEFINED', name='deviceinfoorigin')
def upgrade():
deviceinfoorigin.create(op.get_bind())
op.add_column('device_infos', sa.Column('address_origin_enum', sa.Enum('UNKNOWN', 'OGN_DDB', 'FLARMNET', 'USER_DEFINED', name='deviceinfoorigin'), nullable=False, server_default='UNKNOWN'))
op.execute("UPDATE device_infos SET address_origin_enum = 'UNKNOWN' WHERE address_origin = 0")
op.execute("UPDATE device_infos SET address_origin_enum = 'OGN_DDB' WHERE address_origin = 1")
op.execute("UPDATE device_infos SET address_origin_enum = 'FLARMNET' WHERE address_origin = 2")
op.execute("UPDATE device_infos SET address_origin_enum = 'USER_DEFINED' WHERE address_origin = 3")
op.drop_column('device_infos', 'address_origin')
op.alter_column('device_infos', 'address_origin_enum', new_column_name='address_origin')
def downgrade():
op.add_column('device_infos', sa.Column('address_origin_int', sa.SmallInteger))
op.execute("UPDATE device_infos SET address_origin_int = 0 WHERE address_origin = 'UNKNOWN'")
op.execute("UPDATE device_infos SET address_origin_int = 1 WHERE address_origin = 'OGN_DDB'")
op.execute("UPDATE device_infos SET address_origin_int = 2 WHERE address_origin = 'FLARMNET'")
op.execute("UPDATE device_infos SET address_origin_int = 3 WHERE address_origin = 'USER_DEFINED'")
op.drop_column('device_infos', 'address_origin')
op.alter_column('device_infos', 'address_origin_int', new_column_name='address_origin')
deviceinfoorigin.drop(op.get_bind())

2
setup.cfg
Wyświetl plik

@ -1,2 +1,2 @@
[flake8]
ignore = F401, F841, E402, E501, E126
ignore = F401, F841, E402, E501, E126, E265

11
setup.py
Wyświetl plik

@ -26,8 +26,11 @@ setup(
'Topic :: Scientific/Engineering :: GIS',
'License :: OSI Approved :: GNU Affero General Public License v3',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9-dev'
],
keywords='gliding ogn',
packages=find_packages(exclude=['tests', 'tests.*']),
@ -39,7 +42,8 @@ setup(
'Flask-WTF==0.14.3',
'Flask-Caching==1.9.0',
'geopy==2.0.0',
'celery==5.0.2',
'celery==4.4.7',
'Flask-Redis==0.4.0',
'redis==3.5.3',
'aerofiles==1.0.0',
'geoalchemy2==0.8.4',
@ -50,7 +54,8 @@ setup(
'xmlunittest==0.5.0',
'flower==0.9.5',
'tqdm==4.51.0',
'requests==2.25.0',
'requests==2.25.0',
'matplotlib==3.3.3'
],
test_require=[
'pytest==5.0.1',

0
tests/backend/__init__.py
Wyświetl plik

194
tests/backend/test_backends.py
Wyświetl plik

@ -1,194 +0,0 @@
import json
from datetime import datetime, date
import unittest
from unittest import mock
from xmlunittest import XmlTestMixin
from tests.base import TestBaseDB, db
from app.model import AircraftBeacon, AircraftType, Receiver, Device, DeviceInfo, ReceiverCoverage
from app.backend.liveglidernet import rec, lxml
from app.backend.ognrange import stations2_filtered_pl, max_tile_mgrs_pl
class TestDB(TestBaseDB, XmlTestMixin):
def setUp(self):
super().setUp()
# Prepare Beacons
self.r01 = Receiver(name="Koenigsdf", location_wkt="0101000020E610000061E8FED7A6EE26407F20661C10EA4740", lastseen="2017-12-20 10:00:00", altitude=601, version="0.2.5", platform="ARM")
self.r02 = Receiver(name="Bene", location_wkt="0101000020E6100000D5E76A2BF6C72640D4063A6DA0DB4740", lastseen="2017-12-20 09:45:00", altitude=609, version="0.2.7", platform="x64")
self.r03 = Receiver(name="Ohlstadt", location_wkt="0101000020E6100000057E678EBF772640A142883E32D44740", lastseen="2017-12-20 10:05:00", altitude=655, version="0.2.6", platform="ARM")
db.session.add(self.r01)
db.session.add(self.r02)
db.session.add(self.r03)
db.session.commit()
self.d01 = Device(address="DD4711", lastseen="2017-12-20 10:00:02")
self.d02 = Device(address="DD0815", lastseen="2017-12-20 09:56:00")
db.session.add(self.d01)
db.session.add(self.d02)
db.session.commit()
self.di01 = DeviceInfo(registration="D-4711", competition="Hi", tracked=True, identified=True)
db.session.add(self.di01)
db.session.commit()
self.ab11 = AircraftBeacon(
name="FLRDD4711",
receiver_name="Koenigsdf",
location_wkt="0101000020E6100000211FF46C56ED26402650D7EDC6E94740",
aircraft_type=AircraftType.GLIDER_OR_MOTOR_GLIDER,
timestamp="2017-12-20 10:00:01",
track=105,
ground_speed=57,
climb_rate=-0.5,
)
self.ab12 = AircraftBeacon(
name="FLRDD4711",
receiver_name="Koenigsdf",
location_wkt="0101000020E6100000806DEA295FED2640347D898BB6E94740",
aircraft_type=AircraftType.GLIDER_OR_MOTOR_GLIDER,
timestamp="2017-12-20 10:00:02",
track=123,
ground_speed=55,
climb_rate=-0.4,
altitude=209,
)
self.ab21 = AircraftBeacon(
name="FLRDD0815",
receiver_name="Koenigsdf",
location_wkt="0101000020E6100000F38B25BF58F22640448B6CE7FBE94740",
aircraft_type=AircraftType.POWERED_AIRCRAFT,
timestamp="2017-12-20 09:54:30",
track=280,
ground_speed=80,
climb_rate=-2.9,
)
self.ab22 = AircraftBeacon(
name="FLRDD0815",
receiver_name="Bene",
location_wkt="0101000020E6100000A5E8482EFFF12640DC1EAA16FEE94740",
aircraft_type=AircraftType.POWERED_AIRCRAFT,
timestamp="2017-12-20 09:56:00",
track=270,
ground_speed=77,
climb_rate=-1.5,
altitude=543,
)
db.session.add(self.ab11)
db.session.add(self.ab12)
db.session.add(self.ab21)
db.session.add(self.ab22)
db.session.commit()
self.rc11 = ReceiverCoverage(
location_mgrs_short="32TPU8312", date=date(2017, 12, 20), max_signal_quality=10, max_altitude=1000, min_altitude=600, aircraft_beacon_count=20, device_count=2, receiver=self.r01
)
self.rc12 = ReceiverCoverage(
location_mgrs_short="32TPU8434", date=date(2017, 12, 20), max_signal_quality=10, max_altitude=1000, min_altitude=600, aircraft_beacon_count=20, device_count=2, receiver=self.r01
)
self.rc12 = ReceiverCoverage(
location_mgrs_short="32TPU8434", date=date(2017, 12, 21), max_signal_quality=10, max_altitude=1000, min_altitude=600, aircraft_beacon_count=20, device_count=2, receiver=self.r01
)
self.rc21 = ReceiverCoverage(
location_mgrs_short="32TPU8512", date=date(2017, 12, 20), max_signal_quality=10, max_altitude=1000, min_altitude=600, aircraft_beacon_count=20, device_count=2, receiver=self.r02
)
db.session.add(self.rc11)
db.session.add(self.rc12)
db.session.add(self.rc21)
db.session.commit()
@unittest.skip("broken")
def test_rec(self):
data = rec(min_timestamp=datetime(2017, 12, 19, 10, 0), min_online_timestamp=datetime(2017, 12, 20, 10, 0)).encode(encoding="utf-8")
# Check the document
root = self.assertXmlDocument(data)
self.assertXmlNode(root, tag="markers")
self.assertXpathsOnlyOne(root, ('./m[@a="Koenigsdf"]', './m[@a="Bene"]', './m[@a="Ohlstadt"]'))
# Check the complete document
expected = """<?xml version="1.0" encoding="UTF-8"?>
<markers>
<m e="0"/>
<m a="Bene" b="47.7158333" c="11.3905500" d="0"/>
<m a="Koenigsdf" b="47.8286167" c="11.4661167" d="1"/>
<m a="Ohlstadt" b="47.6577833" c="11.2338833" d="1"/>
</markers>
""".encode(
encoding="utf-8"
)
self.assertXmlEquivalentOutputs(data, expected)
@unittest.skip("broken")
def test_lxml(self):
data = lxml().encode(encoding="utf-8")
# Check the complete document
expected = """<?xml version="1.0" encoding="UTF-8"?>
<markers>
<m a="47.8280667,11.4726500,_15,xxDD0815,543,09:56:00,245,270,77,-1.5,8,Bene,0,xxDD0815"/>
<m a="47.8258833,11.4636167,Hi,D-4711,209,10:00:02,3,123,55,-0.4,1,Koenigsdf,DD4711,xxDD4711"/>
</markers>
""".encode(
encoding="utf-8"
)
self.assertXmlEquivalentOutputs(data, expected)
@mock.patch("app.backend.ognrange.datetime")
def test_stations2_filtered_pl(self, datetime_mock):
datetime_mock.utcnow.return_value = datetime(2017, 12, 20, 10, 0)
result = stations2_filtered_pl(start=date(2017, 12, 15), end=date(2017, 12, 25))
data = json.loads(result)
stations = data["stations"]
self.assertEqual(len(stations), 3)
s1 = stations[0]
s2 = stations[1]
s3 = stations[2]
self.assertEqual(s1["s"], "Bene")
self.assertEqual(s1["lt"], 47.7158)
self.assertEqual(s1["lg"], 11.3906)
self.assertEqual(s1["u"], "D") # Down, because last beacon > 10min. ago
self.assertEqual(s1["ut"], "2017-12-20 09:45")
# self.assertEqual(s1["b"], 0)
self.assertEqual(s1["v"], "0.2.7.x64")
self.assertEqual(s2["s"], "Koenigsdf")
self.assertEqual(s2["lt"], 47.8286)
self.assertEqual(s2["lg"], 11.4661)
self.assertEqual(s2["u"], "U")
self.assertEqual(s2["ut"], "2017-12-20 10:00")
# self.assertEqual(s2["b"], 0)
self.assertEqual(s2["v"], "0.2.5.ARM")
self.assertEqual(s3["s"], "Ohlstadt")
def test_max_tile_mgrs_pl(self):
result = max_tile_mgrs_pl(station="Koenigsdf", start=date(2017, 12, 15), end=date(2017, 12, 25), squares="32TPU")
data = json.loads(result)
self.assertEqual(data["t"], "32TPU")
self.assertEqual(data["p"][0], "8312/1")
self.assertEqual(data["p"][1], "8434/2")
result = max_tile_mgrs_pl(station="Bene", start=date(2017, 12, 15), end=date(2017, 12, 25), squares="32TPU")
data = json.loads(result)
self.assertEqual(data["t"], "32TPU")
self.assertEqual(data["p"][0], "8512/1")
if __name__ == "__main__":
unittest.main()

160
tests/base.py
Wyświetl plik

@ -4,19 +4,177 @@ from app import create_app, db
class TestBaseDB(unittest.TestCase):
def setUp(self):
# Create app context
self.app = create_app('testing')
self.app_context = self.app.app_context()
self.app_context.push()
db.session.execute("DROP TABLE IF EXISTS elevation;")
db.session.commit()
db.drop_all()
db.session.commit()
# ... and create them again
db.session.execute("CREATE EXTENSION IF NOT EXISTS postgis;")
db.session.commit()
db.create_all()
db.session.commit()
db.session.execute("CREATE TABLE IF NOT EXISTS elevation (rast raster);")
db.session.commit()
# ... and create TimescaleDB stuff
db.session.execute("CREATE EXTENSION IF NOT EXISTS timescaledb;")
db.session.execute("SELECT create_hypertable('sender_positions', 'reference_timestamp', chunk_time_interval => interval '1 day', if_not_exists => TRUE);")
#db.session.execute("SELECT create_hypertable('sender_statuses', 'reference_timestamp', chunk_time_interval => interval '1 day', if_not_exists => TRUE);")
db.session.execute("SELECT create_hypertable('receiver_positions', 'reference_timestamp', chunk_time_interval => interval '1 day', if_not_exists => TRUE);")
db.session.execute("SELECT create_hypertable('receiver_statuses', 'reference_timestamp', chunk_time_interval => interval '1 day', if_not_exists => TRUE);")
db.session.commit()
def tearDown(self):
db.session.remove()
db.drop_all()
self.app_context.pop()
def insert_airports_and_devices(self):
db.session.execute("INSERT INTO airports(name, location, altitude, style) VALUES('Benediktbeuren','0101000020E6100000D5E76A2BF6C72640D4063A6DA0DB4740',609,4)")
db.session.execute("INSERT INTO airports(name, location, altitude, style) VALUES('Koenigsdorf','0101000020E610000061E8FED7A6EE26407F20661C10EA4740',600,5)")
db.session.execute("INSERT INTO airports(name, location, altitude, style) VALUES('Ohlstadt','0101000020E6100000057E678EBF772640A142883E32D44740',655,5)")
db.session.execute("INSERT INTO airports(name, location, altitude, style) VALUES('Unterbuchen','0101000020E6100000462575029AF8264089F7098D4DE44740',635,3)")
db.session.execute("UPDATE airports SET border = ST_Expand(location, 0.05)")
db.session.execute("INSERT INTO senders(name, address, aircraft_type) VALUES('FLRDDEFF7', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER')")
db.session.execute("INSERT INTO senders(name, address, aircraft_type) VALUES('FLRDDAC7C', 'DDAC7C', 'GLIDER_OR_MOTOR_GLIDER')")
def insert_sender_positions_broken_rope(self):
"""Fill the db with a winch launch where the rope breaks."""
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000009668B61829F12640330E0887F1E94740',604,'2016-07-02 10:47:12','2016-07-02 10:47:12',0,0,0,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000009668B61829F12640330E0887F1E94740',605,'2016-07-02 10:47:32','2016-07-02 10:47:32',0,0,-0.096520193,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000009668B61829F12640330E0887F1E94740',606,'2016-07-02 10:47:52','2016-07-02 10:47:52',0,0,-0.096520193,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000009668B61829F12640330E0887F1E94740',606,'2016-07-02 10:48:12','2016-07-02 10:48:12',0,0,-0.096520193,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000001B2FDD2406F12640E53C762AF3E94740',606,'2016-07-02 10:48:24','2016-07-02 10:48:24',284,51.85598112,0.299720599,0.1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000F594AFDEBBF02640623583E5F5E94740',610,'2016-07-02 10:48:26','2016-07-02 10:48:26',282,88.89596764,4.729489459,-0.2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000001C0DE02D90F026401564F188F7E94740',619,'2016-07-02 10:48:27','2016-07-02 10:48:27',281,94.45196562,10.66294133,-0.3)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000ABF1D24D62F02640E12D90A0F8E94740',632,'2016-07-02 10:48:28','2016-07-02 10:48:28',278,88.89596764,15.59055118,-0.7)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000069FD40CC38F02640C7925F2CF9E94740',650,'2016-07-02 10:48:29','2016-07-02 10:48:29',273,83.33996966,18.90779782,-0.7)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000002709AF4A0FF02640C7925F2CF9E94740',670,'2016-07-02 10:48:30','2016-07-02 10:48:30',272,79.63597101,20.72136144,-0.3)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000007AA85AF8E7EF2640C7925F2CF9E94740',691,'2016-07-02 10:48:31','2016-07-02 10:48:31',269,79.63597101,21.02108204,-0.4)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000068DB43D5C2EF2640E12D90A0F8E94740',712,'2016-07-02 10:48:32','2016-07-02 10:48:32',267,74.07997303,21.62560325,-0.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000EDA16AE19FEF2640FBC8C014F8E94740',728,'2016-07-02 10:48:33','2016-07-02 10:48:33',266,68.52397506,12.36982474,-0.1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000000AFCCE1C7FEF26401564F188F7E94740',733,'2016-07-02 10:48:34','2016-07-02 10:48:34',266,68.52397506,2.21488443,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000275633585EEF26402FFF21FDF6E94740',731,'2016-07-02 10:48:35','2016-07-02 10:48:35',267,68.52397506,-3.916687833,0.2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000015891C3539EF26402FFF21FDF6E94740',726,'2016-07-02 10:48:36','2016-07-02 10:48:36',270,74.07997303,-6.329692659,1.1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000E63FA4DFBEEE264078C1CDCFFAE94740',712,'2016-07-02 10:48:39','2016-07-02 10:48:39',280,88.89596764,-2.611125222,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000004FF9EABD0BEE2640448B6CE7FBE94740',706,'2016-07-02 10:48:43','2016-07-02 10:48:43',256,90.74796697,-0.198120396,-2.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000046B921B3A0ED264003E78C28EDE94740',706,'2016-07-02 10:48:46','2016-07-02 10:48:46',218,92.59996629,-0.198120396,-1.6)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000005C58F3177ED2640900C4C81DFE94740',703,'2016-07-02 10:48:48','2016-07-02 10:48:48',202,96.30396495,-1.402082804,-1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000211FF46C56ED26402650D7EDC6E94740',702,'2016-07-02 10:48:51','2016-07-02 10:48:51',188,100.0079636,0.502921006,-1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000806DEA295FED2640347D898BB6E94740',704,'2016-07-02 10:48:53','2016-07-02 10:48:53',166,100.0079636,0.802641605,-2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000337D898BB6ED26401383C0CAA1E94740',703,'2016-07-02 10:48:56','2016-07-02 10:48:56',133,101.8599629,-1.803403607,-1.7)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000000C05593CE2ED2640FDF675E09CE94740',700,'2016-07-02 10:48:57','2016-07-02 10:48:57',123,103.7119622,-2.611125222,-1.4)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000F0CCF1F778EE26409FA87F2394E94740',693,'2016-07-02 10:49:00','2016-07-02 10:49:00',105,111.1199596,-2.809245618,-0.6)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000C9073D9B55EF2640BD5296218EE94740',687,'2016-07-02 10:49:04','2016-07-02 10:49:04',97,112.9719589,-1.605283211,-0.1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000006F8104C5EF26400C24287E8CE94740',682,'2016-07-02 10:49:06','2016-07-02 10:49:06',97,114.8239582,-2.407924816,-0.2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000A0648535A8F02640F597DD9387E94740',676,'2016-07-02 10:49:10','2016-07-02 10:49:10',97,118.5279569,-1.402082804,0.1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000D70FC48C03F22640621386EE7FE94740',672,'2016-07-02 10:49:16','2016-07-02 10:49:16',97,116.6759575,-1.000762002,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000A72C431CEBF22640CB7F48BF7DE94740',666,'2016-07-02 10:49:20','2016-07-02 10:49:20',84,114.8239582,-1.605283211,-1.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000BFCAA145B6F32640BD5296218EE94740',662,'2016-07-02 10:49:24','2016-07-02 10:49:24',49,111.1199596,-1.203962408,-1.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000074DA40A70DF4264077E09C11A5E94740',659,'2016-07-02 10:49:27','2016-07-02 10:49:27',23,107.4159609,-1.402082804,-1.4)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000009AE3EFF11CF42640347D898BB6E94740',656,'2016-07-02 10:49:29','2016-07-02 10:49:29',4,101.8599629,-0.797561595,-1.8)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000074DA40A70DF426402650D7EDC6E94740',654,'2016-07-02 10:49:31','2016-07-02 10:49:31',347,101.8599629,-1.706883414,-1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000156A4DF38EF3264086EE7F6DEAE94740',649,'2016-07-02 10:49:36','2016-07-02 10:49:36',312,98.15596427,-1.503683007,-1.4)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000FAEDEBC039F32640E53C762AF3E94740',644,'2016-07-02 10:49:38','2016-07-02 10:49:38',295,96.30396495,-3.012446025,-1.2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B04A0F30E0F22640FBC8C014F8E94740',635,'2016-07-02 10:49:40','2016-07-02 10:49:40',284,94.45196562,-5.125730251,-0.7)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000F38B25BF58F22640448B6CE7FBE94740',623,'2016-07-02 10:49:43','2016-07-02 10:49:43',279,92.59996629,-2.809245618,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000A5E8482EFFF12640DC1EAA16FEE94740',617,'2016-07-02 10:49:45','2016-07-02 10:49:45',279,88.89596764,-3.312166624,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000009F17012859F12640F0AAF40003EA4740',607,'2016-07-02 10:49:49','2016-07-02 10:49:49',279,81.48797034,-1.300482601,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000004B5658830AF12640873E323005EA4740',607,'2016-07-02 10:49:51','2016-07-02 10:49:51',278,74.07997303,-0.294640589,-0.1)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000A0648535A8F0264006373FEB07EA4740',605,'2016-07-02 10:49:54','2016-07-02 10:49:54',280,61.11597775,-0.096520193,0.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E6100000C74B378941F02640E88C28ED0DEA4740',604,'2016-07-02 10:49:58','2016-07-02 10:49:58',292,48.15198247,0.101600203,0.4)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E61000001B5A643BDFEF264045DB1EAA16EA4740',604,'2016-07-02 10:50:04','2016-07-02 10:50:04',302,25.92799056,0.203200406,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000042D2948AB3EF264074029A081BEA4740',604,'2016-07-02 10:50:10','2016-07-02 10:50:10',300,5.555997978,0.101600203,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDEFF7', 'Koenigsdf', 'DDEFF7', 'GLIDER_OR_MOTOR_GLIDER','0101000020E610000013AB192CAFEF264074029A081BEA4740',603,'2016-07-02 10:50:16','2016-07-02 10:50:16',0,0,-0.096520193,0)")
db.session.execute("UPDATE sender_positions SET agl = altitude - 602;")
db.session.commit()
def insert_sender_positions_broken_rope_with_stall(self):
"""Here we have a broken rope where the glider passes again the threshold for take off."""
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',597,'2019-04-13 09:20:14','2019-04-13 09:20:14',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',595,'2019-04-13 09:20:23','2019-04-13 09:20:23',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',595,'2019-04-13 09:20:29','2019-04-13 09:20:29',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',591,'2019-04-13 09:21:01','2019-04-13 09:21:01',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',591,'2019-04-13 09:21:02','2019-04-13 09:21:02',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',589,'2019-04-13 09:21:13','2019-04-13 09:21:13',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',589,'2019-04-13 09:21:29','2019-04-13 09:21:29',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',590,'2019-04-13 09:21:48','2019-04-13 09:21:48',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',591,'2019-04-13 09:22:02','2019-04-13 09:22:02',0,0,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B5040FE689EA264091FC62C92FEA4740',592,'2019-04-13 09:22:22','2019-04-13 09:22:22',0,0,0.1016,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000ED0DBE3099EA2640CA32C4B12EEA4740',593,'2019-04-13 09:22:40','2019-04-13 09:22:40',102,25.925552,0.2032,0.60000002)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000026E4839ECDEA26401904560E2DEA4740',594,'2019-04-13 09:22:42','2019-04-13 09:22:42',100,68.517532,0.2032,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000D044D8F0F4EA2640513AB7F62BEA4740',595,'2019-04-13 09:22:43','2019-04-13 09:22:43',101,81.480309,1.91008,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000025396A721EEB2640A00B49532AEA4740',600,'2019-04-13 09:22:44','2019-04-13 09:22:44',100,90.739433,5.6337199,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000009E8B4814EEB2640CA41AA3B29EA4740',608,'2019-04-13 09:22:45','2019-04-13 09:22:45',100,88.887611,9.2557602,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000087084327AEB264019133C9827EA4740',620,'2019-04-13 09:22:46','2019-04-13 09:22:46',99,87.035782,12.3698,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000246416B4A3EB264052499D8026EA4740',634,'2019-04-13 09:22:47','2019-04-13 09:22:47',97,83.33213,15.2908,-0.89999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000007958A835CDEB264067E4CDF425EA4740',650,'2019-04-13 09:22:48','2019-04-13 09:22:48',94,79.628487,16.093439,-2.0999999)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000CE4C3AB7F6EB264067E4CDF425EA4740',667,'2019-04-13 09:22:49','2019-04-13 09:22:49',91,75.924835,16.89608,-0.89999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000248613AB19EC264067E4CDF425EA4740',684,'2019-04-13 09:22:50','2019-04-13 09:22:50',91,72.221184,17.20088,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000005C532ACE3EEC264067E4CDF425EA4740',701,'2019-04-13 09:22:51','2019-04-13 09:22:51',90,68.517532,16.89608,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000003FF9C5925FEC264067E4CDF425EA4740',718,'2019-04-13 09:22:52','2019-04-13 09:22:52',91,68.517532,16.19504,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000229F615780EC264052499D8026EA4740',733,'2019-04-13 09:22:53','2019-04-13 09:22:53',89,59.258408,14.28496,-1.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B11D82BD9CEC264052499D8026EA4740',741,'2019-04-13 09:22:54','2019-04-13 09:22:54',89,57.406582,3.62204,0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000789CA223B9EC264067E4CDF425EA4740',736,'2019-04-13 09:22:55','2019-04-13 09:22:55',88,53.70293,-8.3413601,0.89999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B0AE00B9D7EC264052499D8026EA4740',724,'2019-04-13 09:22:56','2019-04-13 09:22:56',89,62.962055,-14.5796,0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000E97B17DCFCEC264052499D8026EA4740',710,'2019-04-13 09:22:57','2019-04-13 09:22:57',92,85.18396,-12.1666,1.8)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000CC2A62EB2CED26408A7FFE6825EA4740',703,'2019-04-13 09:22:58','2019-04-13 09:22:58',96,99.998558,-5.92836,2.0999999)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000936DEA295FED2640B5B55F5124EA4740',701,'2019-04-13 09:22:59','2019-04-13 09:22:59',102,99.998558,0.40132001,2.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000CB10C7BAB8ED2640D95F764F1EEA4740',704,'2019-04-13 09:23:01','2019-04-13 09:23:01',116,92.591263,2.21488,5.6999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000002005593CE2ED2640AE38FBF019EA4740',707,'2019-04-13 09:23:02','2019-04-13 09:23:02',133,88.887611,2.8143201,7.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000925CFE43FAED2640E77D426313EA4740',709,'2019-04-13 09:23:03','2019-04-13 09:23:03',147,88.887611,1.50876,6.9000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000CA65AD8E09EE26404BF9EABD0BEA4740',710,'2019-04-13 09:23:04','2019-04-13 09:23:04',159,88.887611,0.60452002,6.9000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000003DF9EABD0BEE2640448B6CE7FBE94740',709,'2019-04-13 09:23:06','2019-04-13 09:23:06',183,92.591263,-0.79755998,5.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000005917B7D100EE2640CBA145B6F3E94740',707,'2019-04-13 09:23:07','2019-04-13 09:23:07',192,94.443085,-2.1082001,3.3)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000076711B0DE0ED2640A098966BE4E94740',701,'2019-04-13 09:23:09','2019-04-13 09:23:09',196,99.998558,-2.61112,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000AF25E4839EED2640E08D2B1BC3E94740',695,'2019-04-13 09:23:13','2019-04-13 09:23:13',202,105.55404,0.1016,1.5)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000002152DD4931ED2640AF16FEF9A3E94740',696,'2019-04-13 09:23:17','2019-04-13 09:23:17',214,103.70221,-0.39624,2.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000EA62C92F96EC264021BF58F28BE94740',696,'2019-04-13 09:23:21','2019-04-13 09:23:21',236,105.55404,0.1016,2.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000005CC2ABD203EC26404478557A80E94740',694,'2019-04-13 09:23:24','2019-04-13 09:23:24',249,107.40586,-1.2039599,2.0999999)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000004182E2C798EB26402FEC0A907BE94740',690,'2019-04-13 09:23:26','2019-04-13 09:23:26',256,111.10951,-2.2098,2.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000098A1F63EEEA26407DBD9CEC79E94740',685,'2019-04-13 09:23:29','2019-04-13 09:23:29',268,114.81316,-1.00076,1.8)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000D1915CFE43EA2640E11A79337DE94740',684,'2019-04-13 09:23:32','2019-04-13 09:23:32',277,112.96133,-0.79755998,0.89999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000044BE55C4D6E926404478557A80E94740',682,'2019-04-13 09:23:34','2019-04-13 09:23:34',280,114.81316,-2.0065999,0.60000002)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000029ED0DBE30E92640932B1BC389E94740',675,'2019-04-13 09:23:37','2019-04-13 09:23:37',292,118.51682,-1.2039599,2.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000D467FD40CCE826409AA87F2394E94740',675,'2019-04-13 09:23:39','2019-04-13 09:23:39',307,114.81316,0.80264002,4.1999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000D49AE61DA7E826404BC8073D9BE94740',677,'2019-04-13 09:23:40','2019-04-13 09:23:40',316,112.96133,2.0116799,5.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000009CC420B072E826403D9B559FABE94740',680,'2019-04-13 09:23:42','2019-04-13 09:23:42',339,103.70221,1.0058399,5.4000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000002A762AF369E8264019D3728DBCE94740',681,'2019-04-13 09:23:44','2019-04-13 09:23:44',358,96.294907,0.2032,4.1999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000F1F44A5986E82640992A1895D4E94740',679,'2019-04-13 09:23:47','2019-04-13 09:23:47',10,94.443085,-2.2098,0.89999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000007F2E244DA9E826401982BD9CECE94740',671,'2019-04-13 09:23:50','2019-04-13 09:23:50',14,96.294907,-2.2098,0.60000002)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000D3EFCCF1F7E8264099ACB00615EA4740',662,'2019-04-13 09:23:55','2019-04-13 09:23:55',21,103.70221,-2.7127199,1.2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000028B1759646E92640513AB7F62BEA4740',655,'2019-04-13 09:23:58','2019-04-13 09:23:58',40,103.70221,-1.905,4.1999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000009A99999999E9264059B71B5736EA4740',652,'2019-04-13 09:24:00','2019-04-13 09:24:00',60,99.998558,-1.2039599,5.0999999)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000448B6CE7FBE9264091DE96B53AEA4740',649,'2019-04-13 09:24:02','2019-04-13 09:24:02',78,98.146736,-2.5095201,4.1999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000000AA4BA9362EA264091DE96B53AEA4740',643,'2019-04-13 09:24:04','2019-04-13 09:24:04',93,98.146736,-2.8092401,3)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B4958DE1C4EA26402E81BA6E37EA4740',636,'2019-04-13 09:24:06','2019-04-13 09:24:06',100,98.146736,-3.71856,1.2)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000005D6DC5FEB2EB2640B597933D2FEA4740',619,'2019-04-13 09:24:11','2019-04-13 09:24:11',100,94.443085,-3.71856,-0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000005BB1BFEC9EEC2640EEDCDAAF28EA4740',602,'2019-04-13 09:24:16','2019-04-13 09:24:16',98,96.294907,-2.7127199,0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000B003E78C28ED2640A01A2FDD24EA4740',598,'2019-04-13 09:24:19','2019-04-13 09:24:19',98,88.887611,-0.70104003,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000009298966BE4ED26408A8EE4F21FEA4740',597,'2019-04-13 09:24:24','2019-04-13 09:24:24',100,59.258408,-0.096519999,0.30000001)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000075C601E130EE2640EEFAA6C31DEA4740',596,'2019-04-13 09:24:28','2019-04-13 09:24:28',86,25.925552,0,-4.1999998)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000091B1E4174BEE26408A8EE4F21FEA4740',597,'2019-04-13 09:24:31','2019-04-13 09:24:31',66,14.814602,-0.096519999,-3)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000001F27563358EE26402722222222EA4740',597,'2019-04-13 09:24:38','2019-04-13 09:24:38',0,0,0.1016,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000CAFFDAD453EE26402722222222EA4740',598,'2019-04-13 09:24:58','2019-04-13 09:24:58',0,0,0.1016,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E6100000586C9DA551EE26402722222222EA4740',597,'2019-04-13 09:25:18','2019-04-13 09:25:18',0,0,0.1016,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000003098A1F63EE2640EEEBC03923EA4740',596,'2019-04-13 09:25:36','2019-04-13 09:25:36',54,1.8518252,0.1016,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E610000003CDF1F778EE2640A01A2FDD24EA4740',594,'2019-04-13 09:25:48','2019-04-13 09:25:48',76,1.8518252,-0.096519999,0)")
db.session.execute("INSERT INTO sender_positions(name, receiver_name, address, aircraft_type, location, altitude, timestamp, reference_timestamp, track, ground_speed, climb_rate, turn_rate) VALUES('FLRDDAC7C','Koenigsd2','DDAC7C','GLIDER_OR_MOTOR_GLIDER','0101000020E61000001FF46C567DEE2640A01A2FDD24EA4740',593,'2019-04-13 09:25:59','2019-04-13 09:25:59',0,0,-0.096519999,0)")
db.session.execute("UPDATE sender_positions SET agl = altitude - 602;")
db.session.commit()
if __name__ == "__main__":
unittest.main()

34
tests/collect/test_database.py
Wyświetl plik

@ -1,34 +0,0 @@
import unittest
from tests.base import TestBaseDB, db
from app.model import AircraftBeacon
from app.collect.database import upsert
class TestDatabase(TestBaseDB):
@unittest.skip("wip")
def test_insert_duplicate_beacons(self):
row1 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:51:00", "ground_speed": None}
row2 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:52:00", "ground_speed": 0}
row3 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:53:00", "ground_speed": 1}
row4 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:54:00", "ground_speed": None}
upsert(session=db.session, model=AircraftBeacon, rows=[row1, row2, row3, row4], update_cols=["ground_speed"])
row5 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:51:00", "ground_speed": 2}
row6 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:52:00", "ground_speed": 3}
row7 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:53:00", "ground_speed": None}
row8 = {"name": "FLRDD0815", "receiver_name": "Koenigsdf", "timestamp": "2019-01-26 11:54:00", "ground_speed": None}
upsert(session=db.session, model=AircraftBeacon, rows=[row5, row6, row7, row8], update_cols=["ground_speed"])
result = db.session.query(AircraftBeacon).order_by(AircraftBeacon.timestamp).all()
self.assertEqual(result[0].ground_speed, 2)
self.assertEqual(result[1].ground_speed, 3)
self.assertEqual(result[2].ground_speed, 1)
self.assertEqual(result[3].ground_speed, None)
if __name__ == "__main__":
unittest.main()

68
tests/collect/test_logbook.py
Wyświetl plik

@ -3,8 +3,8 @@ import unittest
from tests.base import TestBaseDB, db
from app.model import Logbook, Airport, Device, TakeoffLanding
from app.collect.logbook import update_entries
from app.model import Logbook, Airport, Sender, TakeoffLanding
from app.collect.logbook import update_logbook
class TestLogbook(TestBaseDB):
@ -12,8 +12,8 @@ class TestLogbook(TestBaseDB):
super().setUp()
# Create basic data and insert
self.dd0815 = Device(address="DD0815")
self.dd4711 = Device(address="DD4711")
self.dd0815 = Sender(name="FLRDD0815", address="DD0815")
self.dd4711 = Sender(name="FLRDD4711", address="DD4711")
self.koenigsdorf = Airport(name="Koenigsdorf")
self.ohlstadt = Airport(name="Ohlstadt")
@ -26,79 +26,63 @@ class TestLogbook(TestBaseDB):
db.session.commit()
# Prepare takeoff and landings
self.takeoff_koenigsdorf_dd0815 = TakeoffLanding(is_takeoff=True, timestamp="2016-06-01 10:00:00", airport_id=self.koenigsdorf.id, device_id=self.dd0815.id)
self.landing_koenigsdorf_dd0815 = TakeoffLanding(is_takeoff=False, timestamp="2016-06-01 10:05:00", airport_id=self.koenigsdorf.id, device_id=self.dd0815.id)
self.landing_koenigsdorf_dd0815_later = TakeoffLanding(is_takeoff=False, timestamp="2016-06-02 10:05:00", airport_id=self.koenigsdorf.id, device_id=self.dd0815.id)
self.takeoff_ohlstadt_dd4711 = TakeoffLanding(is_takeoff=True, timestamp="2016-06-01 10:00:00", airport_id=self.ohlstadt.id, device_id=self.dd4711.id)
self.takeoff_koenigsdorf_dd0815 = TakeoffLanding(is_takeoff=True, timestamp="2016-06-01 10:00:00", airport_id=self.koenigsdorf.id, sender_id=self.dd0815.id)
self.landing_koenigsdorf_dd0815 = TakeoffLanding(is_takeoff=False, timestamp="2016-06-01 10:05:00", airport_id=self.koenigsdorf.id, sender_id=self.dd0815.id)
self.landing_koenigsdorf_dd0815_later = TakeoffLanding(is_takeoff=False, timestamp="2016-06-02 10:05:00", airport_id=self.koenigsdorf.id, sender_id=self.dd0815.id)
self.takeoff_ohlstadt_dd4711 = TakeoffLanding(is_takeoff=True, timestamp="2016-06-01 10:00:00", airport_id=self.ohlstadt.id, sender_id=self.dd4711.id)
def get_logbook_entries(self):
return db.session.query(Logbook).order_by(Logbook.takeoff_airport_id, Logbook.reftime).all()
return db.session.query(Logbook).order_by(Logbook.takeoff_airport_id, Logbook.reference_timestamp).all()
def test_single_takeoff(self):
db.session.add(self.takeoff_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[0].landing_airport_id, None)
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
entries2 = self.get_logbook_entries()
self.assertEqual(entries, entries2)
def test_single_landing(self):
db.session.add(self.landing_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, None)
self.assertEqual(entries[0].landing_airport_id, self.koenigsdorf.id)
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
entries2 = self.get_logbook_entries()
self.assertEqual(entries, entries2)
def test_different_takeoffs(self):
db.session.add(self.takeoff_koenigsdorf_dd0815)
db.session.add(self.takeoff_ohlstadt_dd4711)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 2)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[1].takeoff_airport_id, self.ohlstadt.id)
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
entries2 = self.get_logbook_entries()
self.assertEqual(entries, entries2)
def test_takeoff_and_landing(self):
db.session.add(self.takeoff_koenigsdorf_dd0815)
db.session.add(self.landing_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[0].landing_airport_id, self.koenigsdorf.id)
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
entries2 = self.get_logbook_entries()
self.assertEqual(entries, entries2)
@unittest.skip('needs information about airport timezone')
def test_takeoff_and_landing_on_different_days(self):
db.session.add(self.takeoff_koenigsdorf_dd0815)
db.session.add(self.landing_koenigsdorf_dd0815_later)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_entries(session=db.session, date=datetime.date(2016, 6, 2))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 2)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
@ -106,15 +90,11 @@ class TestLogbook(TestBaseDB):
self.assertEqual(entries[1].landing_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[1].reftime, self.landing_koenigsdorf_dd0815_later.timestamp)
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
entries2 = self.get_logbook_entries()
self.assertEqual(entries, entries2)
def test_update(self):
db.session.add(self.takeoff_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook(0)
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
@ -122,7 +102,7 @@ class TestLogbook(TestBaseDB):
db.session.add(self.landing_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
@ -131,7 +111,7 @@ class TestLogbook(TestBaseDB):
db.session.add(self.takeoff_ohlstadt_dd4711)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook(0)
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 2)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
@ -141,26 +121,22 @@ class TestLogbook(TestBaseDB):
db.session.add(self.landing_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, None)
self.assertEqual(entries[0].landing_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[0].reftime, self.landing_koenigsdorf_dd0815.timestamp)
self.assertEqual(entries[0].reference_timestamp, self.landing_koenigsdorf_dd0815.timestamp)
db.session.add(self.takeoff_koenigsdorf_dd0815)
db.session.commit()
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
update_logbook()
entries = self.get_logbook_entries()
self.assertEqual(len(entries), 1)
self.assertEqual(entries[0].takeoff_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[0].landing_airport_id, self.koenigsdorf.id)
self.assertEqual(entries[0].reftime, self.takeoff_koenigsdorf_dd0815.timestamp)
update_entries(session=db.session, date=datetime.date(2016, 6, 1))
entries2 = self.get_logbook_entries()
self.assertEqual(entries, entries2)
self.assertEqual(entries[0].reference_timestamp, self.takeoff_koenigsdorf_dd0815.timestamp)
if __name__ == "__main__":

53
tests/collect/test_ognrange.py
Wyświetl plik

@ -1,53 +0,0 @@
from datetime import date
import unittest
from tests.base import TestBaseDB, db
from app.model import AircraftBeacon, Receiver, ReceiverCoverage, Device
from app.collect.ognrange import update_entries
class TestOGNrange(TestBaseDB):
def setUp(self):
super().setUp()
# Create basic data and insert
self.dd0815 = Device(address="DD0815")
self.dd4711 = Device(address="DD4711")
self.r01 = Receiver(name="Koenigsdf")
self.r02 = Receiver(name="Bene")
db.session.add(self.dd0815)
db.session.add(self.dd4711)
db.session.add(self.r01)
db.session.add(self.r02)
db.session.commit()
# Create beacons and insert
self.ab01 = AircraftBeacon(
name="FLRDD0815", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:00", location_mgrs_short="89ABC1267", altitude=800
)
self.ab02 = AircraftBeacon(
name="FLRDD0815", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:01", location_mgrs_short="89ABC1267", altitude=850
)
db.session.add(self.ab01)
db.session.add(self.ab02)
db.session.commit()
@unittest.skip('stats will replaced by timescaledb aggregates')
def test_update_receiver_coverage(self):
update_entries(db.session, date=date(2017, 12, 10))
coverages = db.session.query(ReceiverCoverage).all()
self.assertEqual(len(coverages), 1)
coverage = coverages[0]
self.assertEqual(coverage.location_mgrs_short, "89ABC1267")
self.assertEqual(coverage.receiver_id, self.r01.id)
self.assertEqual(coverage.min_altitude, 800)
self.assertEqual(coverage.max_altitude, 850)
if __name__ == "__main__":
unittest.main()

199
tests/collect/test_stats.py
Wyświetl plik

@ -1,199 +0,0 @@
from datetime import datetime, date
import unittest
from tests.base import TestBaseDB, db
from app.model import AircraftBeacon, ReceiverBeacon, Receiver, Device, DeviceStats
from app.collect.stats import create_device_stats
class TestStats(TestBaseDB):
def setUp(self):
super().setUp()
# Prepare Beacons
self.ab01 = AircraftBeacon(name="FLRDD4711", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:01")
self.ab02 = AircraftBeacon(name="FLRDD4711", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:02")
self.ab03 = AircraftBeacon(name="FLRDD4711", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:03")
self.ab04 = AircraftBeacon(name="FLRDD4711", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:04")
self.ab05 = AircraftBeacon(name="FLRDD4711", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:05")
self.ab06 = AircraftBeacon(name="FLRDD4711", receiver_name="Koenigsdf", timestamp="2017-12-10 10:00:05")
self.rb01 = ReceiverBeacon(name="Koenigsdf", receiver_name="GLIDERN1", timestamp="2017-12-10 09:55:00", altitude=601)
self.rb02 = ReceiverBeacon(name="Koenigsdf", receiver_name="GLIDERN1", timestamp="2017-12-10 10:00:00", altitude=601)
self.rb03 = ReceiverBeacon(name="Koenigsdf", receiver_name="GLIDERN1", timestamp="2017-12-10 10:05:00", altitude=601)
self.r01 = Receiver(name="Koenigsdf")
self.r02 = Receiver(name="Bene")
self.d01 = Device(address="DD4711")
db.session.add(self.r01)
db.session.add(self.d01)
db.session.commit()
@unittest.skip('stats will replaced by timescaledb aggregates')
def test_create_device_stats(self):
# Compute 1st beacon
self.ab01.device = self.d01
self.ab01.receiver = self.r01
db.session.add(self.ab01)
db.session.commit()
today = date(2017, 12, 10)
create_device_stats(db.session, date=today)
devicestats = db.session.query(DeviceStats).all()
self.assertEqual(len(devicestats), 1)
self.assertEqual(devicestats[0].device, self.d01)
self.assertEqual(devicestats[0].max_altitude, None)
self.assertEqual(devicestats[0].receiver_count, 1)
self.assertEqual(devicestats[0].aircraft_beacon_count, 1)
self.assertEqual(devicestats[0].date, datetime.strptime("2017-12-10", "%Y-%m-%d").date())
self.assertEqual(devicestats[0].firstseen, datetime(2017, 12, 10, 10, 0, 1))
self.assertEqual(devicestats[0].lastseen, datetime(2017, 12, 10, 10, 0, 1))
self.assertEqual(devicestats[0].aircraft_type, None)
self.assertEqual(devicestats[0].stealth, None)
self.assertEqual(devicestats[0].software_version, None)
self.assertEqual(devicestats[0].hardware_version, None)
self.assertEqual(devicestats[0].real_address, None)
# Compute 2nd beacon: set altitude, aircraft_type and stealth
self.ab02.device = self.d01
self.ab02.receiver = self.r01
self.ab02.altitude = 200
self.ab02.aircraft_type = 3
self.ab02.stealth = False
db.session.add(self.ab02)
db.session.commit()
create_device_stats(db.session, date=today)
devicestats = db.session.query(DeviceStats).all()
self.assertEqual(len(devicestats), 1)
self.assertEqual(devicestats[0].device, self.d01)
self.assertEqual(devicestats[0].max_altitude, 200)
self.assertEqual(devicestats[0].receiver_count, 1)
self.assertEqual(devicestats[0].aircraft_beacon_count, 2)
self.assertEqual(devicestats[0].date, datetime.strptime("2017-12-10", "%Y-%m-%d").date())
self.assertEqual(devicestats[0].firstseen, datetime(2017, 12, 10, 10, 0, 1))
self.assertEqual(devicestats[0].lastseen, datetime(2017, 12, 10, 10, 0, 2))
self.assertEqual(devicestats[0].aircraft_type, 3)
self.assertEqual(devicestats[0].stealth, False)
self.assertEqual(devicestats[0].software_version, None)
self.assertEqual(devicestats[0].hardware_version, None)
self.assertEqual(devicestats[0].real_address, None)
# Compute 3rd beacon: changed software version, but with error_count > 0
self.ab03.device = self.d01
self.ab03.receiver = self.r01
self.ab03.error_count = 1
self.ab03.software_version = 6.01
db.session.add(self.ab03)
db.session.commit()
create_device_stats(db.session, date=today)
devicestats = db.session.query(DeviceStats).all()
self.assertEqual(len(devicestats), 1)
self.assertEqual(devicestats[0].device, self.d01)
self.assertEqual(devicestats[0].max_altitude, 200)
self.assertEqual(devicestats[0].receiver_count, 1)
self.assertEqual(devicestats[0].aircraft_beacon_count, 2)
self.assertEqual(devicestats[0].date, datetime.strptime("2017-12-10", "%Y-%m-%d").date())
self.assertEqual(devicestats[0].firstseen, datetime(2017, 12, 10, 10, 0, 1))
self.assertEqual(devicestats[0].lastseen, datetime(2017, 12, 10, 10, 0, 2))
self.assertEqual(devicestats[0].aircraft_type, 3)
self.assertEqual(devicestats[0].stealth, False)
self.assertEqual(devicestats[0].software_version, None)
self.assertEqual(devicestats[0].hardware_version, None)
self.assertEqual(devicestats[0].real_address, None)
# Compute 4. beacon: another receiver, greater altitude, software_version, hardware_version, real_address
self.ab04.device = self.d01
self.ab04.receiver = self.r02
self.ab04.altitude = 250
self.ab04.software_version = 6.01
self.ab04.hardware_version = 15
self.ab04.real_address = "DDALFA"
db.session.add(self.ab04)
db.session.commit()
create_device_stats(db.session, date=today)
devicestats = db.session.query(DeviceStats).all()
self.assertEqual(len(devicestats), 1)
self.assertEqual(devicestats[0].device, self.d01)
self.assertEqual(devicestats[0].max_altitude, 250)
self.assertEqual(devicestats[0].receiver_count, 2)
self.assertEqual(devicestats[0].aircraft_beacon_count, 3)
self.assertEqual(devicestats[0].date, datetime.strptime("2017-12-10", "%Y-%m-%d").date())
self.assertEqual(devicestats[0].firstseen, datetime(2017, 12, 10, 10, 0, 1))
self.assertEqual(devicestats[0].lastseen, datetime(2017, 12, 10, 10, 0, 4))
self.assertEqual(devicestats[0].aircraft_type, 3)
self.assertEqual(devicestats[0].stealth, False)
self.assertEqual(devicestats[0].software_version, 6.01)
self.assertEqual(devicestats[0].hardware_version, 15)
self.assertEqual(devicestats[0].real_address, "DDALFA")
# Compute 5. beacon: lower altitude, stealth
self.ab05.device = self.d01
self.ab05.receiver = self.r02
self.ab05.altitude = 100
self.ab05.stealth = True
db.session.add(self.ab05)
db.session.commit()
create_device_stats(db.session, date=today)
devicestats = db.session.query(DeviceStats).all()
self.assertEqual(len(devicestats), 1)
self.assertEqual(devicestats[0].device, self.d01)
self.assertEqual(devicestats[0].max_altitude, 250)
self.assertEqual(devicestats[0].receiver_count, 2)
self.assertEqual(devicestats[0].aircraft_beacon_count, 4)
self.assertEqual(devicestats[0].date, datetime.strptime("2017-12-10", "%Y-%m-%d").date())
self.assertEqual(devicestats[0].firstseen, datetime(2017, 12, 10, 10, 0, 1))
self.assertEqual(devicestats[0].lastseen, datetime(2017, 12, 10, 10, 0, 5))
self.assertEqual(devicestats[0].aircraft_type, 3)
self.assertEqual(devicestats[0].stealth, True)
self.assertEqual(devicestats[0].software_version, 6.01)
self.assertEqual(devicestats[0].hardware_version, 15)
self.assertEqual(devicestats[0].real_address, "DDALFA")
# Compute 6. beacon: beacon from past, greater altitude, newer version
self.ab06.device = self.d01
self.ab06.receiver = self.r02
self.ab06.timestamp = datetime(2017, 12, 10, 9, 59, 50)
self.ab06.altitude = 300
self.ab06.software_version = 6.02
db.session.add(self.ab06)
db.session.commit()
create_device_stats(db.session, date=today)
devicestats = db.session.query(DeviceStats).all()
self.assertEqual(len(devicestats), 1)
self.assertEqual(devicestats[0].device, self.d01)
self.assertEqual(devicestats[0].max_altitude, 300)
self.assertEqual(devicestats[0].receiver_count, 2)
self.assertEqual(devicestats[0].aircraft_beacon_count, 5)
self.assertEqual(devicestats[0].date, datetime.strptime("2017-12-10", "%Y-%m-%d").date())
self.assertEqual(devicestats[0].firstseen, datetime(2017, 12, 10, 9, 59, 50))
self.assertEqual(devicestats[0].lastseen, datetime(2017, 12, 10, 10, 0, 5))
self.assertEqual(devicestats[0].aircraft_type, 3)
self.assertEqual(devicestats[0].stealth, True)
self.assertEqual(devicestats[0].software_version, 6.01)
self.assertEqual(devicestats[0].hardware_version, 15)
self.assertEqual(devicestats[0].real_address, "DDALFA")
if __name__ == "__main__":
unittest.main()

Some files were not shown because too many files have changed in this diff Show More