diff --git a/.travis.yml b/.travis.yml index c16c0e6..fe2dc08 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,21 +1,39 @@ sudo: false language: python +services: + - redis + - postgresql + +addons: + postgresql: "11.2" + +before_install: + - sudo service postgresql stop + - sudo apt-get update + - sudo apt-get remove -y postgresql\* + - sudo apt-get install postgresql-11-postgis-2.5 postgresql-client + - sudo sed -i 's/port = 5433/port = 5432/' /etc/postgresql/11/main/postgresql.conf + - sudo cp /etc/postgresql/{9.6,11}/main/pg_hba.conf + - sudo service postgresql start 11 + install: + - pip install --upgrade pip - pip install tox before_script: + - sudo add-apt-repository ppa:timescale/timescaledb-ppa -y + - sudo apt-get update -q + - sudo apt-get install -y timescaledb-postgresql-11 timescaledb-tools + - sudo timescaledb-tune -yes -pg-version 11 + - sudo service postgresql restart 11 - psql -U postgres -c 'CREATE DATABASE ogn_test;' - psql -U postgres -c 'CREATE EXTENSION postgis;' script: - tox -addons: - postgresql: 9.6 - apt: - packages: - - postgresql-9.6-postgis-2.4 + matrix: include: @@ -28,3 +46,8 @@ matrix: after_success: - tox -e codecov + +after_failure: + - echo "Job failed..." + - sudo cat /var/log/syslog + - psql --version diff --git a/README.md b/README.md index 523aa0c..45cb622 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ It requires [PostgreSQL](http://www.postgresql.org/), [PostGIS](http://www.postg 4. Install [PostgreSQL](http://www.postgresql.org/) with [PostGIS](http://www.postgis.net/) and [TimescaleDB](https://www.timescale.com) Extension. Create a database (use "ogn" as default, otherwise you have to modify the configuration, see below) -5. Optional: Install redis for asynchronous tasks (like takeoff/landing-detection) +5. Install redis for asynchronous tasks (like database feeding, takeoff/landing-detection, ...) ``` apt-get install redis-server @@ -53,13 +53,7 @@ It requires [PostgreSQL](http://www.postgresql.org/), [PostGIS](http://www.postg ./flask database init ``` -8. Optional: Prepare tables for TimescaleDB - - ``` - ./flask database init_timescaledb - ``` - -9. Optional: Import world border dataset (needed if you want to know the country a receiver belongs to, etc.) +8. Optional: Import world border dataset (needed if you want to know the country a receiver belongs to, etc.) Get the [World Borders Dataset](http://thematicmapping.org/downloads/world_borders.php) and unpack it. Then import it into your database (we use "ogn" as database name). @@ -69,40 +63,30 @@ It requires [PostgreSQL](http://www.postgresql.org/), [PostGIS](http://www.postg psql -d ogn -c "DROP TABLE world_borders_temp;" ``` -10. Get world elevation data (needed for AGL calculation) +9. Get world elevation data (needed for AGL calculation) Sources: There are many sources for DEM data. It is important that the spatial reference system (SRID) is the same as the database which is 4326. The [GMTED2010 Viewer](https://topotools.cr.usgs.gov/gmted_viewer/viewer.htm) provides data for the world with SRID 4326. Just download the data you need. - - For Europe we can get the DEM as GeoTIFF files from the [European Environment Agency](https://land.copernicus.eu/imagery-in-situ/eu-dem/eu-dem-v1.1). - Because the SRID of these files is 3035 and we want 4326 we have to convert them (next step) -11. Optional: Convert the elevation data into correct SRID - - We convert elevation from one SRID (here: 3035) to target SRID (4326): + +10. Import the GeoTIFF into the elevation table: ``` - gdalwarp -s_srs "EPSG:3035" -t_srs "EPSG:4326" source.tif target.tif - ``` - -12. Import the GeoTIFF into the elevation table: - - ``` - raster2pgsql -s 4326 -c -C -I -M -t 100x100 elevation_data.tif public.elevation | psql -d ogn + raster2pgsql *.tif -s 4326 -d -M -C -I -F -t 25x25 public.elevation | psql -d ogn ``` -13. Import Airports (needed for takeoff and landing calculation). A cup file is provided under tests: +11. Import Airports (needed for takeoff and landing calculation). A cup file is provided under tests: ``` flask database import_airports tests/SeeYou.cup ``` -14. Import DDB (needed for registration signs in the logbook). +12. Import DDB (needed for registration signs in the logbook). ``` flask database import_ddb ``` -15. Optional: Use supervisord +13. Optional: Use supervisord You can use [Supervisor](http://supervisord.org/) to control the complete system. In the directory deployment/supervisor we have some configuration files to feed the database (ogn-feed), run the celery worker (celeryd), the celery beat (celerybeatd), the celery monitor (flower), and the python wsgi server (gunicorn). All files assume that @@ -128,13 +112,13 @@ The following scripts run in the foreground and should be deamonized - Start a task server (make sure redis is up and running) ``` - celery -A app.collect worker -l info + celery -A celery_app worker -l info ``` - Start the task scheduler (make sure a task server is up and running) ``` - celery -A app.collect beat -l info + celery -A celery_app beat -l info ``` ### Flask - Command Line Interface @@ -173,14 +157,10 @@ Most commands are command groups, so if you execute this command you will get fu ### Available tasks -- `app.collect.celery.update_takeoff_landings` - Compute takeoffs and landings. -- `app.collect.celery.update_logbook_entries` - Add/update logbook entries. -- `app.collect.celery.update_logbook_max_altitude` - Add max altitudes in logbook when flight is complete (takeoff and landing). -- `app.collect.celery.import_ddb` - Import registered devices from the DDB. -- `app.collect.celery.update_receivers_country_code` - Update country code in receivers table if None. -- `app.collect.celery.purge_old_data` - Delete AircraftBeacons and ReceiverBeacons older than given 'age'. -- `app.collect.celery.update_stats` - Create stats and update receivers/devices with stats. -- `app.collect.celery.update_ognrange` - Create receiver coverage stats for Melissas ognrange. +- `app.tasks.update_takeoff_landings` - Compute takeoffs and landings. +- `app.tasks.celery.update_logbook_entries` - Add/update logbook entries. +- `app.tasks.celery.update_logbook_max_altitude` - Add max altitudes in logbook when flight is complete (takeoff and landing). +- `app.tasks.celery.import_ddb` - Import registered devices from the DDB. If the task server is up and running, tasks could be started manually. Here we compute takeoffs and landings for the past 90 minutes: @@ -190,5 +170,18 @@ python3 >>>update_takeoff_landings.delay(last_minutes=90) ``` +or directly from command line: + +``` +celery -A celery_app call takeoff_landings +``` + +## Notes for Raspberry Pi +For matplotlib we need several apt packages installed: + +``` +apt install libatlas3-base libopenjp2-7 libtiff5 +``` + ## License Licensed under the [AGPLv3](LICENSE). diff --git a/app/__init__.py b/app/__init__.py index 374c159..c594b83 100644 --- a/app/__init__.py +++ b/app/__init__.py @@ -1,36 +1,61 @@ +import os + from flask import Flask from flask_bootstrap import Bootstrap from flask_sqlalchemy import SQLAlchemy from flask_migrate import Migrate from flask_caching import Cache from celery import Celery +from flask_redis import FlaskRedis + +from config import configs bootstrap = Bootstrap() db = SQLAlchemy() migrate = Migrate() cache = Cache() -celery = Celery(__name__, broker='redis://localhost:6379/0') +redis_client = FlaskRedis() +celery = Celery() -def create_app(config_name='development'): +def create_app(config_name='default'): # Initialize Flask app = Flask(__name__) # Load the configuration - if config_name == 'testing': - app.config.from_object('app.config.test') - else: - app.config.from_object('app.config.default') + configuration = configs[config_name] + app.config.from_object(configuration) app.config.from_envvar("OGN_CONFIG_MODULE", silent=True) - celery.config_from_object(app.config) # Initialize other things bootstrap.init_app(app) db.init_app(app) migrate.init_app(app, db) cache.init_app(app) + redis_client.init_app(app) + init_celery(app) + register_blueprints(app) + + return app + + +def register_blueprints(app): from app.main import bp as bp_main app.register_blueprint(bp_main) - return app + +def init_celery(app=None): + app = app or create_app(os.getenv('FLASK_CONFIG') or 'default') + celery.conf.broker_url = app.config['BROKER_URL'] + celery.conf.result_backend = app.config['CELERY_RESULT_BACKEND'] + celery.conf.update(app.config) + + class ContextTask(celery.Task): + """Make celery tasks work with Flask app context""" + def __call__(self, *args, **kwargs): + with app.app_context(): + return self.run(*args, **kwargs) + + celery.Task = ContextTask + return celery diff --git a/app/backend/liveglidernet.py b/app/backend/liveglidernet.py deleted file mode 100644 index 3dd7a77..0000000 --- a/app/backend/liveglidernet.py +++ /dev/null @@ -1,103 +0,0 @@ -from datetime import datetime, timezone - -from app.model import AircraftBeacon, ReceiverBeacon -from app import db - - -def utc_to_local(utc_dt): - return utc_dt.replace(tzinfo=timezone.utc).astimezone(tz=None) - - -def encode(address): - return "xx" + address - - -def decode(code): - return code[2:9] - - -def rec(min_timestamp, min_online_timestamp): - last_seen_query = ( - db.session.query(ReceiverBeacon).filter(ReceiverBeacon.timestamp > min_timestamp).order_by(ReceiverBeacon.receiver_id, ReceiverBeacon.timestamp).distinct(ReceiverBeacon.receiver_id) - ) - - lines = [] - lines.append('') - lines.append("") - lines.append('') - for receiver_beacon in last_seen_query: - if receiver_beacon.location is None or receiver_beacon.name.startswith("FNB"): - continue - lines.append( - ''.format( - receiver_beacon.name, receiver_beacon.location.latitude, receiver_beacon.location.longitude, receiver_beacon.timestamp < min_online_timestamp - ) - ) - - lines.append("") - xml = "\n".join(lines) - - return xml - - -def lxml(show_offline=False, lat_max=90, lat_min=-90, lon_max=180, lon_min=-180): - - timestamp_range_filter = [db.between(AircraftBeacon.timestamp, datetime(2018, 7, 31, 11, 55, 0), datetime(2018, 7, 31, 12, 5, 0))] - - last_seen_query = db.session.query(AircraftBeacon).filter(*timestamp_range_filter).order_by(AircraftBeacon.device_id, AircraftBeacon.timestamp).distinct(AircraftBeacon.device_id) - lines = list() - lines.append('') - lines.append("") - - for aircraft_beacon in last_seen_query: - device = aircraft_beacon.device - - code = encode(device.address) - - if device.info: - if not device.info.tracked or not device.info.identified: - continue - - if not device.info.competition: - competition = device.info.registration[-2:] - else: - competition = device.info.competition - - if not device.info.registration: - registration = "???" - else: - registration = device.info.registration - - address = device.address - - else: - competition = ("_" + code[-2:]).lower() - registration = code - address = 0 - - elapsed_time = datetime.utcnow() - aircraft_beacon.timestamp - elapsed_seconds = int(elapsed_time.total_seconds()) - - lines.append( - ' '.format( - aircraft_beacon.location.latitude, - aircraft_beacon.location.longitude, - competition, - registration, - int(aircraft_beacon.altitude), - utc_to_local(aircraft_beacon.timestamp).strftime("%H:%M:%S"), - elapsed_seconds, - int(aircraft_beacon.track), - int(aircraft_beacon.ground_speed), - int(aircraft_beacon.climb_rate * 10) / 10, - aircraft_beacon.aircraft_type, - aircraft_beacon.receiver_name, - address, - code, - ) - ) - - lines.append("") - xml = "\n".join(lines) - - return xml diff --git a/app/backend/ognrange.py b/app/backend/ognrange.py index cdef59b..37bd4c1 100644 --- a/app/backend/ognrange.py +++ b/app/backend/ognrange.py @@ -1,8 +1,6 @@ import json from datetime import datetime, timedelta -from sqlalchemy import func, case -from sqlalchemy.sql.expression import label from app.model import Receiver, ReceiverCoverage from app import db @@ -26,11 +24,11 @@ def stations2_filtered_pl(start, end): query = ( db.session.query( Receiver.name.label("s"), - label("lt", func.round(func.ST_Y(Receiver.location_wkt) * 10000) / 10000), - label("lg", func.round(func.ST_X(Receiver.location_wkt) * 10000) / 10000), - case([(Receiver.lastseen > last_10_minutes, "U")], else_="D").label("u"), + db.label("lt", db.func.round(db.func.ST_Y(Receiver.location_wkt) * 10000) / 10000), + db.label("lg", db.func.round(db.func.ST_X(Receiver.location_wkt) * 10000) / 10000), + db.case([(Receiver.lastseen > last_10_minutes, "U")], else_="D").label("u"), Receiver.lastseen.label("ut"), - label("v", Receiver.version + "." + Receiver.platform), + db.label("v", Receiver.version + "." + Receiver.platform), ) .order_by(Receiver.lastseen) .filter(db.or_(db.and_(start < Receiver.firstseen, end > Receiver.firstseen), db.and_(start < Receiver.lastseen, end > Receiver.lastseen))) @@ -44,10 +42,10 @@ def stations2_filtered_pl(start, end): def max_tile_mgrs_pl(station, start, end, squares): query = ( - db.session.query(func.right(ReceiverCoverage.location_mgrs_short, 4), func.count(ReceiverCoverage.location_mgrs_short)) + db.session.query(db.func.right(ReceiverCoverage.location_mgrs_short, 4), db.func.count(ReceiverCoverage.location_mgrs_short)) .filter(db.and_(Receiver.id == ReceiverCoverage.receiver_id, Receiver.name == station)) .filter(ReceiverCoverage.location_mgrs_short.like(squares + "%")) - .group_by(func.right(ReceiverCoverage.location_mgrs_short, 4)) + .group_by(db.func.right(ReceiverCoverage.location_mgrs_short, 4)) ) res = {"t": squares, "p": ["{}/{}".format(r[0], r[1]) for r in query.all()]} diff --git a/app/collect/celery.py b/app/collect/celery.py deleted file mode 100644 index 1bb4c2f..0000000 --- a/app/collect/celery.py +++ /dev/null @@ -1,106 +0,0 @@ -import datetime - -from celery.utils.log import get_task_logger - -from app.collect.takeoff_landings import update_entries as takeoff_update_entries - -from app.collect.logbook import update_entries as logbook_update_entries -from app.collect.logbook import update_max_altitudes as logbook_update_max_altitudes - -from app.collect.database import import_ddb as device_infos_import_ddb -from app.collect.database import update_country_code as receivers_update_country_code - -from app.collect.stats import create_device_stats, update_device_stats_jumps, create_receiver_stats, create_relation_stats, update_qualities, update_receivers, update_devices - -from app.collect.ognrange import update_entries as receiver_coverage_update_entries - -from app import db -from app import celery - - -logger = get_task_logger(__name__) - - -@celery.task(name="update_takeoff_landings") -def update_takeoff_landings(last_minutes): - """Compute takeoffs and landings.""" - - end = datetime.datetime.utcnow() - start = end - datetime.timedelta(minutes=last_minutes) - result = takeoff_update_entries(session=db.session, start=start, end=end, logger=logger) - return result - - -@celery.task(name="update_logbook_entries") -def update_logbook_entries(day_offset): - """Add/update logbook entries.""" - - date = datetime.datetime.today() + datetime.timedelta(days=day_offset) - result = logbook_update_entries(session=db.session, date=date, logger=logger) - return result - - -@celery.task(name="update_logbook_max_altitude") -def update_logbook_max_altitude(day_offset): - """Add max altitudes in logbook when flight is complete (takeoff and landing).""" - - date = datetime.datetime.today() + datetime.timedelta(days=day_offset) - result = logbook_update_max_altitudes(session=db.session, date=date, logger=logger) - return result - - -@celery.task(name="import_ddb") -def import_ddb(): - """Import registered devices from the DDB.""" - - result = device_infos_import_ddb(session=db.session, logger=logger) - return result - - -@celery.task(name="update_receivers_country_code") -def update_receivers_country_code(): - """Update country code in receivers table if None.""" - - result = receivers_update_country_code(session=db.session, logger=logger) - return result - - -@celery.task(name="purge_old_data") -def purge_old_data(max_hours): - """Delete AircraftBeacons and ReceiverBeacons older than given 'age'.""" - - from app.model import AircraftBeacon, ReceiverBeacon - - min_timestamp = datetime.datetime.utcnow() - datetime.timedelta(hours=max_hours) - aircraft_beacons_deleted = db.session.query(AircraftBeacon).filter(AircraftBeacon.timestamp < min_timestamp).delete() - - receiver_beacons_deleted = db.session.query(ReceiverBeacon).filter(ReceiverBeacon.timestamp < min_timestamp).delete() - - db.session.commit() - - result = "{} AircraftBeacons deleted, {} ReceiverBeacons deleted".format(aircraft_beacons_deleted, receiver_beacons_deleted) - return result - - -@celery.task(name="update_stats") -def update_stats(day_offset): - """Create stats and update receivers/devices with stats.""" - - date = datetime.datetime.today() + datetime.timedelta(days=day_offset) - - create_device_stats(session=db.session, date=date) - update_device_stats_jumps(session=db.session, date=date) - create_receiver_stats(session=db.session, date=date) - create_relation_stats(session=db.session, date=date) - update_qualities(session=db.session, date=date) - update_receivers(session=db.session) - update_devices(session=db.session) - - -@celery.task(name="update_ognrange") -def update_ognrange(day_offset): - """Create receiver coverage stats for Melissas ognrange.""" - - date = datetime.datetime.today() + datetime.timedelta(days=day_offset) - - receiver_coverage_update_entries(session=db.session, date=date) diff --git a/app/collect/database.py b/app/collect/database.py index 6193a92..0c5d51b 100644 --- a/app/collect/database.py +++ b/app/collect/database.py @@ -1,12 +1,12 @@ -from sqlalchemy.sql import null, and_, func, case from sqlalchemy.dialects.postgresql import insert from flask import current_app -from app.model import Country, DeviceInfo, DeviceInfoOrigin, Receiver +from app import db +from app.model import SenderInfo, SenderInfoOrigin, Receiver from app.utils import get_ddb, get_flarmnet -def upsert(session, model, rows, update_cols): +def upsert(model, rows, update_cols): """Insert rows in model. On conflicting update columns if new value IS NOT NULL.""" table = model.__table__ @@ -14,59 +14,40 @@ def upsert(session, model, rows, update_cols): stmt = insert(table).values(rows) on_conflict_stmt = stmt.on_conflict_do_update( - index_elements=table.primary_key.columns, set_={k: case([(getattr(stmt.excluded, k) != null(), getattr(stmt.excluded, k))], else_=getattr(model, k)) for k in update_cols} + index_elements=table.primary_key.columns, set_={k: db.case([(getattr(stmt.excluded, k) != db.null(), getattr(stmt.excluded, k))], else_=getattr(model, k)) for k in update_cols} ) # print(compile_query(on_conflict_stmt)) - session.execute(on_conflict_stmt) + return on_conflict_stmt -def update_device_infos(session, address_origin, path=None): - if address_origin == DeviceInfoOrigin.FLARMNET: +def update_device_infos(address_origin, path=None): + if address_origin == SenderInfoOrigin.FLARMNET: device_infos = get_flarmnet(fln_file=path) else: device_infos = get_ddb(csv_file=path) - session.query(DeviceInfo).filter(DeviceInfo.address_origin == address_origin).delete(synchronize_session="fetch") - session.commit() + db.session.query(SenderInfo).filter(SenderInfo.address_origin == address_origin).delete(synchronize_session="fetch") + db.session.commit() for device_info in device_infos: device_info.address_origin = address_origin - session.bulk_save_objects(device_infos) - session.commit() + db.session.bulk_save_objects(device_infos) + db.session.commit() return len(device_infos) -def import_ddb(session, logger=None): +def import_ddb(logger=None): """Import registered devices from the DDB.""" if logger is None: logger = current_app.logger logger.info("Import registered devices fom the DDB...") - counter = update_device_infos(session, DeviceInfoOrigin.OGN_DDB) + counter = update_device_infos(SenderInfoOrigin.OGN_DDB) - finish_message = "DeviceInfo: {} inserted.".format(counter) - logger.info(finish_message) - return finish_message - - -def update_country_code(session, logger=None): - """Update country code in receivers table if None.""" - - if logger is None: - logger = current_app.logger - - update_receivers = ( - session.query(Receiver) - .filter(and_(Receiver.country_id == null(), Receiver.location_wkt != null(), func.st_within(Receiver.location_wkt, Country.geom))) - .update({Receiver.country_id: Country.gid}, synchronize_session="fetch") - ) - - session.commit() - - finish_message = "Receivers (country): {} updated".format(update_receivers) + finish_message = "SenderInfo: {} inserted.".format(counter) logger.info(finish_message) return finish_message diff --git a/app/collect/flights.py b/app/collect/flights.py new file mode 100644 index 0000000..4e79e58 --- /dev/null +++ b/app/collect/flights.py @@ -0,0 +1,116 @@ +from datetime import date + +from app import db + +NOTHING = "" +CONTEST_RELEVANT = "AND agl < 1000" +LOW_PASS = "AND agl < 50 and ground_speed > 250" + + +def compute_flights(date, flight_type=0): + if flight_type == 0: + filter = NOTHING + elif flight_type == 1: + filter = CONTEST_RELEVANT + elif flight_type == 2: + filter = LOW_PASS + + date_str = date.strftime("%Y-%m-%d") + + query = f""" + INSERT INTO flights(date, sender_id, flight_type, multilinestring, simple_multilinestring) + SELECT '{date_str}' AS date, + s.id AS sender_id, + {flight_type} as flight_type, + st_collect(sq5.linestring order BY sq5.part) multilinestring, + st_collect(st_simplify(sq5.linestring, 0.0001) ORDER BY sq5.part) simple_multilinestring + FROM ( + SELECT sq4.name, + sq4.part, + st_makeline(sq4.location ORDER BY sq4.timestamp) AS linestring + FROM ( + SELECT sq3.timestamp, + sq3.location, + sq3.name, + SUM(sq3.ping) OVER (partition BY sq3.name ORDER BY sq3.timestamp) AS part + FROM ( + SELECT sq2.t1 AS timestamp, + sq2.l1 AS location, + sq2.s1 AS name, + CASE + WHEN sq2.s1 = sq2.s2 AND sq2.t1 - sq2.t2 < interval'100s' AND ST_DistanceSphere(sq2.l1, sq2.l2) < 1000 THEN 0 + ELSE 1 + END AS ping + FROM ( + SELECT sq.timestamp t1, + lag(sq.timestamp) OVER (partition BY sq.name ORDER BY sq.timestamp) t2, + sq.location l1, + lag(sq.location) OVER (partition BY sq.name ORDER BY sq.timestamp) l2, + sq.name s1, + lag(sq.name) OVER (partition BY sq.name ORDER BY sq.timestamp) s2 + FROM ( + SELECT DISTINCT ON (name, timestamp) name, timestamp, location + FROM sender_positions + WHERE reference_timestamp BETWEEN '{date_str} 00:00:00' AND '{date_str} 23:59:59' {filter} + ORDER BY name, timestamp, error_count + ) AS sq + ) AS sq2 + ) AS sq3 + ) AS sq4 + GROUP BY sq4.name, sq4.part + ) AS sq5 + INNER JOIN senders AS s ON sq5.name = s.name + GROUP BY s.id + ON CONFLICT DO NOTHING; + """ + + db.session.execute(query) + db.session.commit() + + +def compute_gaps(date): + date_str = date.strftime("%Y-%m-%d") + + query = f""" + INSERT INTO flights(date, flight_type, sender_id, multilinestring) + SELECT '{date_str}' AS date, + 3 AS flight_type, + s.id AS sender_id, + ST_Collect(sq3.path) + FROM ( + SELECT sq2.s1 AS name, + ST_MakeLine(sq2.l1, sq2.l2) AS path + FROM + ( + SELECT sq.timestamp t1, + LAG(sq.timestamp) OVER (PARTITION BY sq.timestamp::DATE, sq.name ORDER BY sq.timestamp) t2, + sq.location l1, + LAG(sq.location) OVER (PARTITION BY sq.timestamp::DATE, sq.name ORDER BY sq.timestamp) l2, + sq.name s1, + LAG(sq.name) OVER (PARTITION BY sq.timestamp::DATE, sq.name ORDER BY sq.timestamp) s2 + FROM + ( + SELECT DISTINCT ON (name, timestamp) name, timestamp, location, agl + FROM sender_positions + WHERE reference_timestamp BETWEEN '{date_str} 00:00:00' AND '{date_str} 23:59:59' AND agl > 300 + ORDER BY name, timestamp, error_count + ) AS sq + ) AS sq2 + WHERE EXTRACT(epoch FROM sq2.t1 - sq2.t2) > 300 + AND ST_DistanceSphere(sq2.l1, sq2.l2) / EXTRACT(epoch FROM sq2.t1 - sq2.t2) BETWEEN 15 AND 50 + ) AS sq3 + INNER JOIN senders AS s on sq3.name = s.name + GROUP BY s.id + ON CONFLICT DO NOTHING; + """ + + db.session.execute(query) + db.session.commit() + + +if __name__ == '__main__': + from app import create_app + app = create_app() + with app.app_context(): + result = compute_flights(date=date(2020, 10, 28)) + print(result) diff --git a/app/collect/gateway.py b/app/collect/gateway.py new file mode 100644 index 0000000..3cb47e7 --- /dev/null +++ b/app/collect/gateway.py @@ -0,0 +1,24 @@ +from datetime import datetime +from flask import current_app + +from app import redis_client +from app.gateway.message_handling import sender_position_csv_strings_to_db, receiver_position_csv_strings_to_db, receiver_status_csv_strings_to_db + + +def transfer_from_redis_to_database(): + def unmapping(string): + return string[0].decode('utf-8') + + receiver_status_data = list(map(unmapping, redis_client.zpopmin('receiver_status', 100000))) + receiver_position_data = list(map(unmapping, redis_client.zpopmin('receiver_position', 100000))) + sender_status_data = list(map(unmapping, redis_client.zpopmin('sender_status', 100000))) + sender_position_data = list(map(unmapping, redis_client.zpopmin('sender_position', 100000))) + + receiver_status_csv_strings_to_db(lines=receiver_status_data) + receiver_position_csv_strings_to_db(lines=receiver_position_data) + sender_position_csv_strings_to_db(lines=sender_position_data) + + current_app.logger.debug(f"transfer_from_redis_to_database: rx_stat: {len(receiver_status_data):6d}\trx_pos: {len(receiver_position_data):6d}\ttx_stat: {len(sender_status_data):6d}\ttx_pos: {len(sender_position_data):6d}") + + finish_message = f"Database: {len(receiver_status_data)+len(receiver_position_data)+len(sender_status_data)+len(sender_position_data)} inserted" + return finish_message diff --git a/app/collect/logbook.py b/app/collect/logbook.py index 86ae1ad..128dde4 100644 --- a/app/collect/logbook.py +++ b/app/collect/logbook.py @@ -1,199 +1,382 @@ -from sqlalchemy import and_, or_, insert, update, exists, between -from sqlalchemy.sql import func, null -from sqlalchemy.sql.expression import true, false +from sqlalchemy.dialects.postgresql import insert # special insert for upsert ("ON CONFLICT ...") from flask import current_app -from app.model import TakeoffLanding, Logbook, AircraftBeacon +from app.model import Airport, Country, SenderPosition, Sender, TakeoffLanding, Logbook from app.utils import date_to_timestamps +from datetime import datetime, timedelta -def update_entries(session, date, logger=None): - """Add/update logbook entries.""" +from app import db - if logger is None: - logger = current_app.logger - logger.info("Compute logbook.") +# takeoff / landing detection is based on 3 consecutive points +MIN_TAKEOFF_SPEED = 55 # takeoff detection: 1st point below, 2nd and 3rd above this limit +MAX_LANDING_SPEED = 40 # landing detection: 1st point above, 2nd and 3rd below this limit +MIN_TAKEOFF_CLIMB_RATE = -5 # takeoff detection: glider should not sink too much +MAX_LANDING_SINK_RATE = 5 # landing detection: glider should not climb too much +MAX_EVENT_DURATION = 100 # the points must not exceed this duration +MAX_EVENT_RADIUS = 5000 # the points must not exceed this radius around the 2nd point +MAX_EVENT_AGL = 200 # takeoff / landing must not exceed this altitude AGL - # limit time range to given date and set window partition and window order - (start, end) = date_to_timestamps(date) - pa = TakeoffLanding.device_id - wo = and_(TakeoffLanding.device_id, TakeoffLanding.airport_id, TakeoffLanding.timestamp) - # make a query with current, previous and next "takeoff_landing" event, so we can find complete flights +def update_takeoff_landings(start, end): + """Compute takeoffs and landings.""" + + current_app.logger.info("Compute takeoffs and landings.") + + # considered time interval should not exceed a complete day + if end - start > timedelta(days=1): + abort_message = "TakeoffLanding: timeinterval start='{}' and end='{}' is too big.".format(start, end) + current_app.logger.warn(abort_message) + return abort_message + + # check if we have any airport + airports_query = db.session.query(Airport).limit(1) + if not airports_query.all(): + abort_message = "TakeoffLanding: Cannot calculate takeoff and landings without any airport! Please import airports first." + current_app.logger.warn(abort_message) + return abort_message + + # get beacons for selected time range (+ buffer for duration), one per name and timestamp sq = ( - session.query( - TakeoffLanding.device_id, - func.lag(TakeoffLanding.device_id).over(partition_by=pa, order_by=wo).label("device_id_prev"), - func.lead(TakeoffLanding.device_id).over(partition_by=pa, order_by=wo).label("device_id_next"), - TakeoffLanding.timestamp, - func.lag(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_prev"), - func.lead(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_next"), - TakeoffLanding.track, - func.lag(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_prev"), - func.lead(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_next"), - TakeoffLanding.is_takeoff, - func.lag(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_prev"), - func.lead(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_next"), - TakeoffLanding.airport_id, - func.lag(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_prev"), - func.lead(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_next"), - ) - .filter(between(TakeoffLanding.timestamp, start, end)) + db.session.query(SenderPosition.name, SenderPosition.timestamp, SenderPosition.location, SenderPosition.track, db.func.coalesce(SenderPosition.ground_speed, 0.0).label("ground_speed"), SenderPosition.altitude, db.func.coalesce(SenderPosition.climb_rate, 0.0).label("climb_rate")) + .distinct(SenderPosition.name, SenderPosition.timestamp) + .order_by(SenderPosition.name, SenderPosition.timestamp, SenderPosition.error_count) + .filter(SenderPosition.agl <= MAX_EVENT_AGL) + .filter(db.between(SenderPosition.reference_timestamp, start - timedelta(seconds=MAX_EVENT_DURATION), end + timedelta(seconds=MAX_EVENT_DURATION))) .subquery() ) - # find complete flights - complete_flight_query = session.query( - sq.c.timestamp.label("reftime"), - sq.c.device_id.label("device_id"), - sq.c.timestamp.label("takeoff_timestamp"), - sq.c.track.label("takeoff_track"), - sq.c.airport_id.label("takeoff_airport_id"), - sq.c.timestamp_next.label("landing_timestamp"), - sq.c.track_next.label("landing_track"), - sq.c.airport_id_next.label("landing_airport_id"), - ).filter(and_(sq.c.is_takeoff == true(), sq.c.is_takeoff_next == false())) + # make a query with current, previous and next position + sq2 = db.session.query( + sq.c.name, + db.func.lag(sq.c.name).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("name_prev"), + db.func.lead(sq.c.name).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("name_next"), + sq.c.timestamp, + db.func.lag(sq.c.timestamp).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("timestamp_prev"), + db.func.lead(sq.c.timestamp).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("timestamp_next"), + sq.c.location, + db.func.lag(sq.c.location).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("location_wkt_prev"), + db.func.lead(sq.c.location).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("location_wkt_next"), + sq.c.track, + db.func.lag(sq.c.track).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("track_prev"), + db.func.lead(sq.c.track).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("track_next"), + sq.c.ground_speed, + db.func.lag(sq.c.ground_speed).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("ground_speed_prev"), + db.func.lead(sq.c.ground_speed).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("ground_speed_next"), + sq.c.altitude, + db.func.lag(sq.c.altitude).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("altitude_prev"), + db.func.lead(sq.c.altitude).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("altitude_next"), + sq.c.climb_rate, + db.func.lag(sq.c.climb_rate).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("climb_rate_prev"), + db.func.lead(sq.c.climb_rate).over(partition_by=sq.c.name, order_by=sq.c.timestamp).label("climb_rate_next"), + ).subquery() - # find landings without start + # consider only positions between start and end and with predecessor and successor and limit distance and duration between points + sq3 = ( + db.session.query(sq2) + .filter(db.and_(sq2.c.name_prev != db.null(), sq2.c.name_next != db.null())) + .filter(db.and_(db.func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_prev) < MAX_EVENT_RADIUS, db.func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_next) < MAX_EVENT_RADIUS)) + .filter(sq2.c.timestamp_next - sq2.c.timestamp_prev < timedelta(seconds=MAX_EVENT_DURATION)) + .filter(db.between(sq2.c.timestamp, start, end)) + .subquery() + ) + + # find possible takeoffs and landings + sq4 = ( + db.session.query( + sq3.c.timestamp, + db.case( + [ + (sq3.c.ground_speed > MIN_TAKEOFF_SPEED, sq3.c.location_wkt_prev), # on takeoff we take the location from the previous fix because it is nearer to the airport + (sq3.c.ground_speed <= MIN_TAKEOFF_SPEED, sq3.c.location), + ] + ).label("location"), + db.case([(sq3.c.ground_speed > MAX_LANDING_SPEED, sq3.c.track), (sq3.c.ground_speed <= MAX_LANDING_SPEED, sq3.c.track_prev)]).label( + "track" + ), # on landing we take the track from the previous fix because gliders tend to leave the runway quickly + sq3.c.ground_speed, + sq3.c.altitude, + db.case([(sq3.c.ground_speed > MIN_TAKEOFF_SPEED, True), (sq3.c.ground_speed < MAX_LANDING_SPEED, False)]).label("is_takeoff"), + sq3.c.name, + ) + .filter( + db.or_( + db.and_(sq3.c.ground_speed_prev < MIN_TAKEOFF_SPEED, sq3.c.ground_speed > MIN_TAKEOFF_SPEED, sq3.c.ground_speed_next > MIN_TAKEOFF_SPEED, sq3.c.climb_rate > MIN_TAKEOFF_CLIMB_RATE), # takeoff + db.and_(sq3.c.ground_speed_prev > MAX_LANDING_SPEED, sq3.c.ground_speed < MAX_LANDING_SPEED, sq3.c.ground_speed_next < MAX_LANDING_SPEED, sq3.c.climb_rate < MAX_LANDING_SINK_RATE), # landing + ) + ) + .subquery() + ) + + # get the sender id instead of the name and consider them if the are near airports ... + sq5 = ( + db.session.query( + sq4.c.timestamp, sq4.c.track, sq4.c.is_takeoff, Sender.id.label("sender_id"), Airport.id.label("airport_id"), db.func.ST_DistanceSphere(sq4.c.location, Airport.location_wkt).label("airport_distance"), Airport.country_code + ) + .filter(db.and_(db.func.ST_Within(sq4.c.location, Airport.border), + db.between(Airport.style, 2, 5))) + .filter(sq4.c.name == Sender.name) + .subquery() + ) + + # ... and take the nearest airport + sq6 = ( + db.session.query(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.sender_id, sq5.c.airport_id, sq5.c.country_code) + .distinct(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.sender_id) + .order_by(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.sender_id, sq5.c.airport_distance) + .subquery() + ) + + # ... add the country + takeoff_landing_query = ( + db.session.query(sq6.c.timestamp, sq6.c.track, sq6.c.is_takeoff, sq6.c.sender_id, sq6.c.airport_id, Country.gid) + .join(Country, sq6.c.country_code == Country.iso2, isouter=True) + .subquery() + ) + + # ... and save them + ins = insert(TakeoffLanding) \ + .from_select((TakeoffLanding.timestamp, TakeoffLanding.track, TakeoffLanding.is_takeoff, TakeoffLanding.sender_id, TakeoffLanding.airport_id, TakeoffLanding.country_id), takeoff_landing_query) \ + .on_conflict_do_nothing(index_elements=[TakeoffLanding.timestamp, TakeoffLanding.sender_id, TakeoffLanding.airport_id]) + + result = db.session.execute(ins) + db.session.commit() + insert_counter = result.rowcount + + finish_message = "TakeoffLandings: {} inserted".format(insert_counter) + current_app.logger.info(finish_message) + return finish_message + + +def update_logbook(offset_days=None): + """Add/update logbook entries.""" + + current_app.logger.info("Compute logbook.") + + # limit time range to given date and set window partition and window order + if offset_days: + (start, end) = date_to_timestamps(datetime.utcnow() - timedelta(days=offset_days)) + else: + (start, end) = date_to_timestamps(datetime.utcnow().date()) + pa = TakeoffLanding.sender_id + wo = db.and_(TakeoffLanding.sender_id, TakeoffLanding.timestamp, TakeoffLanding.airport_id) + + # make a query with previous, current and next "takeoff_landing" event, so we can find complete flights + sq = ( + db.session.query( + TakeoffLanding.sender_id, + db.func.lag(TakeoffLanding.sender_id).over(partition_by=pa, order_by=wo).label("sender_id_prev"), + db.func.lead(TakeoffLanding.sender_id).over(partition_by=pa, order_by=wo).label("sender_id_next"), + TakeoffLanding.timestamp, + db.func.lag(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_prev"), + db.func.lead(TakeoffLanding.timestamp).over(partition_by=pa, order_by=wo).label("timestamp_next"), + TakeoffLanding.track, + db.func.lag(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_prev"), + db.func.lead(TakeoffLanding.track).over(partition_by=pa, order_by=wo).label("track_next"), + TakeoffLanding.is_takeoff, + db.func.lag(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_prev"), + db.func.lead(TakeoffLanding.is_takeoff).over(partition_by=pa, order_by=wo).label("is_takeoff_next"), + TakeoffLanding.airport_id, + db.func.lag(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_prev"), + db.func.lead(TakeoffLanding.airport_id).over(partition_by=pa, order_by=wo).label("airport_id_next") + ) + .subquery() + ) + + # find (new) starts without landing + only_starts_query = ( + db.session.query( + sq.c.sender_id.label("sender_id"), + sq.c.timestamp.label("takeoff_timestamp"), + sq.c.track.label("takeoff_track"), + sq.c.airport_id.label("takeoff_airport_id") + ) + .filter(sq.c.is_takeoff == db.true()) + .filter(db.or_(sq.c.is_takeoff_next == db.true(), sq.c.is_takeoff_next == db.null())) + .filter(~Logbook.query.filter(db.and_(Logbook.sender_id == sq.c.sender_id, Logbook.takeoff_timestamp == sq.c.timestamp, Logbook.takeoff_airport_id == sq.c.airport_id)).exists()) + ) + ins = insert(Logbook).from_select( + ( + Logbook.sender_id, + Logbook.takeoff_timestamp, + Logbook.takeoff_track, + Logbook.takeoff_airport_id + ), + only_starts_query, + ) + result = db.session.execute(ins) + current_app.logger.debug(f"Added {result.rowcount} starts") + db.session.commit() + + # find (new) landings without start only_landings_query = ( - session.query( - sq.c.timestamp.label("reftime"), - sq.c.device_id.label("device_id"), - null().label("takeoff_timestamp"), - null().label("takeoff_track"), - null().label("takeoff_airport_id"), + db.session.query( + sq.c.sender_id.label("sender_id"), sq.c.timestamp.label("landing_timestamp"), sq.c.track.label("landing_track"), sq.c.airport_id.label("landing_airport_id"), ) - .filter(sq.c.is_takeoff == false()) - .filter(or_(sq.c.is_takeoff_prev == false(), sq.c.is_takeoff_prev == null())) + .filter(db.or_(sq.c.is_takeoff_prev == db.false(), sq.c.is_takeoff_prev == db.null())) + .filter(sq.c.is_takeoff == db.false()) + .filter(~Logbook.query.filter(db.and_(Logbook.sender_id == sq.c.sender_id, Logbook.landing_timestamp == sq.c.timestamp, Logbook.landing_airport_id == sq.c.airport_id)).exists()) ) + ins = insert(Logbook).from_select( + ( + Logbook.sender_id, + Logbook.landing_timestamp, + Logbook.landing_track, + Logbook.landing_airport_id + ), + only_landings_query, + ) + result = db.session.execute(ins) + current_app.logger.debug(f"Added {result.rowcount} landings") + db.session.commit() - # find starts without landing - only_starts_query = ( - session.query( - sq.c.timestamp.label("reftime"), - sq.c.device_id.label("device_id"), + # find complete flights + complete_flight_query = ( + db.session.query( + sq.c.sender_id.label("sender_id"), sq.c.timestamp.label("takeoff_timestamp"), sq.c.track.label("takeoff_track"), sq.c.airport_id.label("takeoff_airport_id"), - null().label("landing_timestamp"), - null().label("landing_track"), - null().label("landing_airport_id"), + sq.c.timestamp_next.label("landing_timestamp"), + sq.c.track_next.label("landing_track"), + sq.c.airport_id_next.label("landing_airport_id"), ) - .filter(sq.c.is_takeoff == true()) - .filter(or_(sq.c.is_takeoff_next == true(), sq.c.is_takeoff_next == null())) + .filter(sq.c.is_takeoff == db.true()) + .filter(sq.c.is_takeoff_next == db.false()) + .subquery() ) - # unite all computated flights - union_query = complete_flight_query.union(only_landings_query, only_starts_query).subquery() - - # if a logbook entry exist --> update it - upd = ( - update(Logbook) - .where( - and_( - Logbook.device_id == union_query.c.device_id, - union_query.c.takeoff_airport_id != null(), - union_query.c.landing_airport_id != null(), - or_( - and_(Logbook.takeoff_airport_id == union_query.c.takeoff_airport_id, Logbook.takeoff_timestamp == union_query.c.takeoff_timestamp, Logbook.landing_airport_id == null()), - and_(Logbook.takeoff_airport_id == null(), Logbook.landing_airport_id == union_query.c.landing_airport_id, Logbook.landing_timestamp == union_query.c.landing_timestamp), - ), - ) - ) - .values( - { - "reftime": union_query.c.reftime, - "takeoff_timestamp": union_query.c.takeoff_timestamp, - "takeoff_track": union_query.c.takeoff_track, - "takeoff_airport_id": union_query.c.takeoff_airport_id, - "landing_timestamp": union_query.c.landing_timestamp, - "landing_track": union_query.c.landing_track, - "landing_airport_id": union_query.c.landing_airport_id, - } - ) + # insert (new) flights + new_flights_query = ( + db.session.query(complete_flight_query) + .filter(~Logbook.query.filter(db.and_(Logbook.sender_id == complete_flight_query.c.sender_id, Logbook.landing_timestamp == complete_flight_query.c.landing_timestamp, Logbook.landing_airport_id == complete_flight_query.c.landing_airport_id)).exists()) + .filter(~Logbook.query.filter(db.and_(Logbook.sender_id == complete_flight_query.c.sender_id, Logbook.takeoff_timestamp == complete_flight_query.c.takeoff_timestamp, Logbook.takeoff_airport_id == complete_flight_query.c.takeoff_airport_id)).exists()) ) - - result = session.execute(upd) - update_counter = result.rowcount - session.commit() - logger.debug("Updated logbook entries: {}".format(update_counter)) - - # if a logbook entry doesnt exist --> insert it - new_logbook_entries = session.query(union_query).filter( - ~exists().where( - and_( - Logbook.device_id == union_query.c.device_id, - or_( - and_(Logbook.takeoff_airport_id == union_query.c.takeoff_airport_id, Logbook.takeoff_timestamp == union_query.c.takeoff_timestamp), - and_(Logbook.takeoff_airport_id == null(), union_query.c.takeoff_airport_id == null()), - ), - or_( - and_(Logbook.landing_airport_id == union_query.c.landing_airport_id, Logbook.landing_timestamp == union_query.c.landing_timestamp), - and_(Logbook.landing_airport_id == null(), union_query.c.landing_airport_id == null()), - ), - ) - ) - ) - ins = insert(Logbook).from_select( ( - Logbook.reftime, - Logbook.device_id, + Logbook.sender_id, Logbook.takeoff_timestamp, Logbook.takeoff_track, Logbook.takeoff_airport_id, Logbook.landing_timestamp, Logbook.landing_track, - Logbook.landing_airport_id, + Logbook.landing_airport_id ), - new_logbook_entries, + new_flights_query ) + result = db.session.execute(ins) + current_app.logger.debug(f"Added {result.rowcount} complete flights") + db.session.commit() - result = session.execute(ins) - insert_counter = result.rowcount - session.commit() + # update existing landing with takeoff from complete flight + upd = db.update(Logbook) \ + .where(db.and_( + Logbook.sender_id == complete_flight_query.c.sender_id, + Logbook.takeoff_timestamp == db.null(), + Logbook.takeoff_airport_id == db.null(), + Logbook.landing_timestamp != db.null(), + Logbook.landing_timestamp == complete_flight_query.c.landing_timestamp, + Logbook.landing_airport_id == complete_flight_query.c.landing_airport_id + )) \ + .values(takeoff_timestamp=complete_flight_query.c.takeoff_timestamp, + takeoff_track=complete_flight_query.c.takeoff_track, + takeoff_airport_id=complete_flight_query.c.takeoff_airport_id) + result = db.session.execute(upd) + current_app.logger.debug(f"Updated {result.rowcount} takeoffs to complete flights") + db.session.commit() - finish_message = "Logbook: {} inserted, {} updated".format(insert_counter, update_counter) - logger.debug(finish_message) - return finish_message + # update existing takeoff with landing from complete flight + upd = db.update(Logbook) \ + .where(db.and_( + Logbook.sender_id == complete_flight_query.c.sender_id, + Logbook.takeoff_timestamp != db.null(), + Logbook.takeoff_timestamp == complete_flight_query.c.takeoff_timestamp, + Logbook.takeoff_airport_id == complete_flight_query.c.takeoff_airport_id, + Logbook.landing_timestamp == db.null(), + Logbook.landing_airport_id == db.null() + )) \ + .values(landing_timestamp=complete_flight_query.c.landing_timestamp, + landing_track=complete_flight_query.c.landing_track, + landing_airport_id=complete_flight_query.c.landing_airport_id) + result = db.session.execute(upd) + current_app.logger.debug(f"Updated {result.rowcount} landings to complete flights") + db.session.commit() + + return -def update_max_altitudes(session, date, logger=None): +def update_max_altitudes(): + MAX_UPDATES = 60 + + query = """ + UPDATE logbooks + SET max_altitude = sq2.max_altitude + FROM ( + SELECT sq.logbook_id, MAX(sp.altitude) AS max_altitude + FROM ( + SELECT + l.id AS logbook_id, s.name, l.takeoff_timestamp, l.landing_timestamp + FROM logbooks AS l + INNER JOIN senders AS s ON l.sender_id = s.id + WHERE + l.takeoff_timestamp IS NOT NULL + AND l.landing_timestamp IS NOT NULL + AND l.max_altitude IS NULL + LIMIT 1 + ) AS sq, + sender_positions AS sp + WHERE sp.reference_timestamp BETWEEN sq.takeoff_timestamp AND sq.landing_timestamp + AND sp.name = sq.name + GROUP BY sq.logbook_id + ) AS sq2 + WHERE logbooks.id = sq2.logbook_id; + """ + + update_counter = 0 + for _ in range(MAX_UPDATES): + result = db.session.execute(query) + db.session.commit() + + return update_counter + + +def update_max_altitudes_orm(): """Add max altitudes in logbook when flight is complete (takeoff and landing).""" - if logger is None: - logger = current_app.logger - - logger.info("Update logbook max altitude.") - - if session is None: - session = current_app.session - - (start, end) = date_to_timestamps(date) + current_app.logger.info("Update logbook max altitude.") logbook_entries = ( - session.query(Logbook.id) - .filter(and_(Logbook.takeoff_timestamp != null(), Logbook.landing_timestamp != null(), Logbook.max_altitude == null())) - .filter(between(Logbook.reftime, start, end)) + db.session.query(Logbook.id, Sender.name) + .filter(db.and_(Logbook.takeoff_timestamp != db.null(), Logbook.landing_timestamp != db.null(), Logbook.max_altitude == db.null())) + .filter(Logbook.sender_id == Sender.id) + .limit(1) .subquery() ) max_altitudes = ( - session.query(Logbook.id, func.max(AircraftBeacon.altitude).label("max_altitude")) - .filter(Logbook.id == logbook_entries.c.id) - .filter(and_(AircraftBeacon.device_id == Logbook.device_id, AircraftBeacon.timestamp >= Logbook.takeoff_timestamp, AircraftBeacon.timestamp <= Logbook.landing_timestamp)) + db.session.query(logbook_entries.c.id, db.func.max(SenderPosition.altitude).label("max_altitude")) + .filter(db.and_(db.between_(SenderPosition.timestamp >= Logbook.takeoff_timestamp, SenderPosition.timestamp <= Logbook.landing_timestamp), SenderPosition.name == logbook_entries.c.name)) .group_by(Logbook.id) .subquery() ) - update_logbook = session.query(Logbook).filter(Logbook.id == max_altitudes.c.id).update({Logbook.max_altitude: max_altitudes.c.max_altitude}, synchronize_session="fetch") + update_logbooks = db.session.query(Logbook).filter(Logbook.id == max_altitudes.c.id).update({Logbook.max_altitude: max_altitudes.c.max_altitude}, synchronize_session="fetch") - session.commit() + db.session.commit() - finish_message = "Logbook (altitude): {} entries updated.".format(update_logbook) - logger.info(finish_message) + finish_message = "Logbook (altitude): {} entries updated.".format(update_logbooks) return finish_message + + +if __name__ == '__main__': + from app import create_app + app = create_app() + with app.app_context(): + result = update_takeoff_landings(start=datetime(2020, 11, 9, 10, 0, 0), end=datetime(2020, 11, 9, 15, 30, 0)) + result = update_logbook() + result = update_max_altitudes_orm() + print(result) diff --git a/app/collect/ognrange.py b/app/collect/ognrange.py deleted file mode 100644 index bca7ab4..0000000 --- a/app/collect/ognrange.py +++ /dev/null @@ -1,104 +0,0 @@ -from sqlalchemy import Date -from sqlalchemy import and_, insert, update, exists, between -from sqlalchemy.sql import func, null -from flask import current_app - -from app.model import AircraftBeacon, Receiver, ReceiverCoverage -from app.utils import date_to_timestamps - - -def update_entries(session, date, logger=None): - """Create receiver coverage stats for Melissas ognrange.""" - - if logger is None: - logger = current_app.logger - - logger.info("Compute receiver coverages.") - - (start, end) = date_to_timestamps(date) - - # Filter aircraft beacons - sq = ( - session.query(AircraftBeacon.location_mgrs_short, AircraftBeacon.receiver_name, AircraftBeacon.signal_quality, AircraftBeacon.altitude, AircraftBeacon.address) - .filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.location_mgrs_short != null(), AircraftBeacon.receiver_name != null(), AircraftBeacon.address != null())) - .subquery() - ) - - # ... and group them by reduced MGRS, receiver and date - sq2 = ( - session.query( - sq.c.location_mgrs_short, - sq.c.receiver_name, - func.cast(date, Date).label("date"), - func.max(sq.c.signal_quality).label("max_signal_quality"), - func.min(sq.c.altitude).label("min_altitude"), - func.max(sq.c.altitude).label("max_altitude"), - func.count(sq.c.altitude).label("aircraft_beacon_count"), - func.count(func.distinct(sq.c.address)).label("device_count"), - ) - .group_by(sq.c.location_mgrs_short, sq.c.receiver_name) - .subquery() - ) - - # Replace receiver_name with receiver_id - sq3 = ( - session.query( - sq2.c.location_mgrs_short, - Receiver.id.label("receiver_id"), - sq2.c.date, - sq2.c.max_signal_quality, - sq2.c.min_altitude, - sq2.c.max_altitude, - sq2.c.aircraft_beacon_count, - sq2.c.device_count, - ) - .filter(sq2.c.receiver_name == Receiver.name) - .subquery() - ) - - # if a receiver coverage entry exist --> update it - upd = ( - update(ReceiverCoverage) - .where(and_(ReceiverCoverage.location_mgrs_short == sq3.c.location_mgrs_short, ReceiverCoverage.receiver_id == sq3.c.receiver_id, ReceiverCoverage.date == date)) - .values( - { - "max_signal_quality": sq3.c.max_signal_quality, - "min_altitude": sq3.c.min_altitude, - "max_altitude": sq3.c.max_altitude, - "aircraft_beacon_count": sq3.c.aircraft_beacon_count, - "device_count": sq3.c.device_count, - } - ) - ) - - result = session.execute(upd) - update_counter = result.rowcount - session.commit() - logger.debug("Updated receiver coverage entries: {}".format(update_counter)) - - # if a receiver coverage entry doesnt exist --> insert it - new_coverage_entries = session.query(sq3).filter( - ~exists().where(and_(ReceiverCoverage.location_mgrs_short == sq3.c.location_mgrs_short, ReceiverCoverage.receiver_id == sq3.c.receiver_id, ReceiverCoverage.date == date)) - ) - - ins = insert(ReceiverCoverage).from_select( - ( - ReceiverCoverage.location_mgrs_short, - ReceiverCoverage.receiver_id, - ReceiverCoverage.date, - ReceiverCoverage.max_signal_quality, - ReceiverCoverage.min_altitude, - ReceiverCoverage.max_altitude, - ReceiverCoverage.aircraft_beacon_count, - ReceiverCoverage.device_count, - ), - new_coverage_entries, - ) - - result = session.execute(ins) - insert_counter = result.rowcount - session.commit() - - finish_message = "ReceiverCoverage: {} inserted, {} updated".format(insert_counter, update_counter) - logger.debug(finish_message) - return finish_message diff --git a/app/collect/stats.py b/app/collect/stats.py deleted file mode 100644 index 8e1d352..0000000 --- a/app/collect/stats.py +++ /dev/null @@ -1,450 +0,0 @@ -from flask import current_app -from sqlalchemy import insert, distinct, between, literal -from sqlalchemy.sql import null, and_, func, or_, update -from sqlalchemy.sql.expression import case - -from app.model import AircraftBeacon, DeviceStats, Country, CountryStats, ReceiverStats, ReceiverBeacon, RelationStats, Receiver, Device -from app.utils import date_to_timestamps - - -# 40dB@10km is enough for 640km -MAX_PLAUSIBLE_QUALITY = 40 - - -def create_device_stats(session, date, logger=None): - """Add/update device stats.""" - - if logger is None: - logger = current_app.logger - - (start, end) = date_to_timestamps(date) - - # First kill the stats for the selected date - deleted_counter = session.query(DeviceStats).filter(DeviceStats.date == date).delete() - - # Since "distinct count" does not work in window functions we need a work-around for receiver counting - sq = ( - session.query(AircraftBeacon, func.dense_rank().over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.receiver_id).label("dr")) - .filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.device_id != null())) - .filter(or_(AircraftBeacon.error_count == 0, AircraftBeacon.error_count == null())) - .subquery() - ) - - # Calculate stats, firstseen, lastseen and last values != NULL - device_stats = session.query( - distinct(sq.c.device_id).label("device_id"), - literal(date).label("date"), - func.max(sq.c.dr).over(partition_by=sq.c.device_id).label("receiver_count"), - func.max(sq.c.altitude).over(partition_by=sq.c.device_id).label("max_altitude"), - func.count(sq.c.device_id).over(partition_by=sq.c.device_id).label("aircraft_beacon_count"), - func.first_value(sq.c.name).over(partition_by=sq.c.device_id, order_by=case([(sq.c.name == null(), None)], else_=sq.c.timestamp).asc().nullslast()).label("name"), - func.first_value(sq.c.timestamp).over(partition_by=sq.c.device_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).asc().nullslast()).label("firstseen"), - func.first_value(sq.c.timestamp).over(partition_by=sq.c.device_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("lastseen"), - func.first_value(sq.c.aircraft_type).over(partition_by=sq.c.device_id, order_by=case([(sq.c.aircraft_type == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("aircraft_type"), - func.first_value(sq.c.stealth).over(partition_by=sq.c.device_id, order_by=case([(sq.c.stealth == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("stealth"), - func.first_value(sq.c.software_version) - .over(partition_by=sq.c.device_id, order_by=case([(sq.c.software_version == null(), None)], else_=sq.c.timestamp).desc().nullslast()) - .label("software_version"), - func.first_value(sq.c.hardware_version) - .over(partition_by=sq.c.device_id, order_by=case([(sq.c.hardware_version == null(), None)], else_=sq.c.timestamp).desc().nullslast()) - .label("hardware_version"), - func.first_value(sq.c.real_address).over(partition_by=sq.c.device_id, order_by=case([(sq.c.real_address == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("real_address"), - ).subquery() - - # And insert them - ins = insert(DeviceStats).from_select( - [ - DeviceStats.device_id, - DeviceStats.date, - DeviceStats.receiver_count, - DeviceStats.max_altitude, - DeviceStats.aircraft_beacon_count, - DeviceStats.name, - DeviceStats.firstseen, - DeviceStats.lastseen, - DeviceStats.aircraft_type, - DeviceStats.stealth, - DeviceStats.software_version, - DeviceStats.hardware_version, - DeviceStats.real_address, - ], - device_stats, - ) - res = session.execute(ins) - insert_counter = res.rowcount - session.commit() - logger.debug("DeviceStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter)) - - return "DeviceStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter) - - -def create_receiver_stats(session, date, logger=None): - """Add/update receiver stats.""" - - if logger is None: - logger = current_app.logger - - (start, end) = date_to_timestamps(date) - - # First kill the stats for the selected date - deleted_counter = session.query(ReceiverStats).filter(ReceiverStats.date == date).delete() - - # Select one day - sq = session.query(ReceiverBeacon).filter(between(ReceiverBeacon.timestamp, start, end)).subquery() - - # Calculate stats, firstseen, lastseen and last values != NULL - receiver_stats = session.query( - distinct(sq.c.receiver_id).label("receiver_id"), - literal(date).label("date"), - func.first_value(sq.c.timestamp).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).asc().nullslast()).label("firstseen"), - func.first_value(sq.c.timestamp).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.timestamp == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("lastseen"), - func.first_value(sq.c.location).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.location == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("location_wkt"), - func.first_value(sq.c.altitude).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.altitude == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("altitude"), - func.first_value(sq.c.version).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.version == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("version"), - func.first_value(sq.c.platform).over(partition_by=sq.c.receiver_id, order_by=case([(sq.c.platform == null(), None)], else_=sq.c.timestamp).desc().nullslast()).label("platform"), - ).subquery() - - # And insert them - ins = insert(ReceiverStats).from_select( - [ - ReceiverStats.receiver_id, - ReceiverStats.date, - ReceiverStats.firstseen, - ReceiverStats.lastseen, - ReceiverStats.location_wkt, - ReceiverStats.altitude, - ReceiverStats.version, - ReceiverStats.platform, - ], - receiver_stats, - ) - res = session.execute(ins) - insert_counter = res.rowcount - session.commit() - logger.warn("ReceiverStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter)) - - # Update aircraft_beacon_count, aircraft_count and max_distance - aircraft_beacon_stats = ( - session.query( - AircraftBeacon.receiver_id, - func.count(AircraftBeacon.timestamp).label("aircraft_beacon_count"), - func.count(func.distinct(AircraftBeacon.device_id)).label("aircraft_count"), - func.max(AircraftBeacon.distance).label("max_distance"), - ) - .filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.error_count == 0, AircraftBeacon.quality <= MAX_PLAUSIBLE_QUALITY, AircraftBeacon.relay == null())) - .group_by(AircraftBeacon.receiver_id) - .subquery() - ) - - upd = ( - update(ReceiverStats) - .where(and_(ReceiverStats.date == date, ReceiverStats.receiver_id == aircraft_beacon_stats.c.receiver_id)) - .values( - {"aircraft_beacon_count": aircraft_beacon_stats.c.aircraft_beacon_count, "aircraft_count": aircraft_beacon_stats.c.aircraft_count, "max_distance": aircraft_beacon_stats.c.max_distance} - ) - ) - - result = session.execute(upd) - update_counter = result.rowcount - session.commit() - logger.warn("Updated {} ReceiverStats".format(update_counter)) - - return "ReceiverStats for {}: {} deleted, {} inserted, {} updated".format(date, deleted_counter, insert_counter, update_counter) - - -def create_country_stats(session, date, logger=None): - if logger is None: - logger = current_app.logger - - (start, end) = date_to_timestamps(date) - - # First kill the stats for the selected date - deleted_counter = session.query(CountryStats).filter(CountryStats.date == date).delete() - - country_stats = ( - session.query(literal(date), Country.gid, func.count(AircraftBeacon.timestamp).label("aircraft_beacon_count"), func.count(func.distinct(AircraftBeacon.receiver_id)).label("device_count")) - .filter(between(AircraftBeacon.timestamp, start, end)) - .filter(func.st_contains(Country.geom, AircraftBeacon.location)) - .group_by(Country.gid) - .subquery() - ) - - # And insert them - ins = insert(CountryStats).from_select([CountryStats.date, CountryStats.country_id, CountryStats.aircraft_beacon_count, CountryStats.device_count], country_stats) - res = session.execute(ins) - insert_counter = res.rowcount - session.commit() - - -def update_device_stats_jumps(session, date, logger=None): - """Update device stats jumps.""" - - if logger is None: - logger = current_app.logger - - (start, end) = date_to_timestamps(date) - - # speed limits in m/s (values above indicates a unplausible position / jump) - max_horizontal_speed = 1000 - max_vertical_speed = 100 - max_jumps = 10 # threshold for an 'ambiguous' device - - # find consecutive positions for a device - sq = ( - session.query( - AircraftBeacon.device_id, - AircraftBeacon.timestamp, - func.lead(AircraftBeacon.timestamp).over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.timestamp).label("timestamp_next"), - AircraftBeacon.location_wkt, - func.lead(AircraftBeacon.location_wkt).over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.timestamp).label("location_next"), - AircraftBeacon.altitude, - func.lead(AircraftBeacon.altitude).over(partition_by=AircraftBeacon.device_id, order_by=AircraftBeacon.timestamp).label("altitude_next"), - ) - .filter(and_(between(AircraftBeacon.timestamp, start, end), AircraftBeacon.error_count == 0)) - .subquery() - ) - - # calc vertial and horizontal speed between points - sq2 = ( - session.query( - sq.c.device_id, - (func.st_distancesphere(sq.c.location_next, sq.c.location) / (func.extract("epoch", sq.c.timestamp_next) - func.extract("epoch", sq.c.timestamp))).label("horizontal_speed"), - ((sq.c.altitude_next - sq.c.altitude) / (func.extract("epoch", sq.c.timestamp_next) - func.extract("epoch", sq.c.timestamp))).label("vertical_speed"), - ) - .filter(and_(sq.c.timestamp != null(), sq.c.timestamp_next != null(), sq.c.timestamp < sq.c.timestamp_next)) - .subquery() - ) - - # ... and find and count 'jumps' - sq3 = ( - session.query(sq2.c.device_id, func.sum(case([(or_(func.abs(sq2.c.horizontal_speed) > max_horizontal_speed, func.abs(sq2.c.vertical_speed) > max_vertical_speed), 1)], else_=0)).label("jumps")) - .group_by(sq2.c.device_id) - .subquery() - ) - - upd = update(DeviceStats).where(and_(DeviceStats.date == date, DeviceStats.device_id == sq3.c.device_id)).values({"ambiguous": sq3.c.jumps > max_jumps, "jumps": sq3.c.jumps}) - - result = session.execute(upd) - update_counter = result.rowcount - session.commit() - logger.warn("Updated {} DeviceStats jumps".format(update_counter)) - - return "DeviceStats jumps for {}: {} updated".format(date, update_counter) - - -def create_relation_stats(session, date, logger=None): - """Add/update relation stats.""" - - if logger is None: - logger = current_app.logger - - (start, end) = date_to_timestamps(date) - - # First kill the stats for the selected date - deleted_counter = session.query(RelationStats).filter(RelationStats.date == date).delete() - - # Calculate stats for selected day - relation_stats = ( - session.query(literal(date), AircraftBeacon.device_id, AircraftBeacon.receiver_id, func.max(AircraftBeacon.quality), func.count(AircraftBeacon.timestamp)) - .filter( - and_( - between(AircraftBeacon.timestamp, start, end), - AircraftBeacon.distance > 1000, - AircraftBeacon.error_count == 0, - AircraftBeacon.quality <= MAX_PLAUSIBLE_QUALITY, - AircraftBeacon.ground_speed > 10, - ) - ) - .group_by(literal(date), AircraftBeacon.device_id, AircraftBeacon.receiver_id) - .subquery() - ) - - # And insert them - ins = insert(RelationStats).from_select([RelationStats.date, RelationStats.device_id, RelationStats.receiver_id, RelationStats.quality, RelationStats.beacon_count], relation_stats) - res = session.execute(ins) - insert_counter = res.rowcount - session.commit() - logger.warn("RelationStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter)) - - return "RelationStats for {}: {} deleted, {} inserted".format(date, deleted_counter, insert_counter) - - -def update_qualities(session, date, logger=None): - """Calculate relative qualities of receivers and devices.""" - - if logger is None: - logger = current_app.logger - - # Calculate avg quality of devices - dev_sq = session.query(RelationStats.device_id, func.avg(RelationStats.quality).label("quality")).filter(RelationStats.date == date).group_by(RelationStats.device_id).subquery() - - dev_upd = update(DeviceStats).where(and_(DeviceStats.date == date, DeviceStats.device_id == dev_sq.c.device_id)).values({"quality": dev_sq.c.quality}) - - dev_result = session.execute(dev_upd) - dev_update_counter = dev_result.rowcount - session.commit() - logger.warn("Updated {} DeviceStats: quality".format(dev_update_counter)) - - # Calculate avg quality of receivers - rec_sq = session.query(RelationStats.receiver_id, func.avg(RelationStats.quality).label("quality")).filter(RelationStats.date == date).group_by(RelationStats.receiver_id).subquery() - - rec_upd = update(ReceiverStats).where(and_(ReceiverStats.date == date, ReceiverStats.receiver_id == rec_sq.c.receiver_id)).values({"quality": rec_sq.c.quality}) - - rec_result = session.execute(rec_upd) - rec_update_counter = rec_result.rowcount - session.commit() - logger.warn("Updated {} ReceiverStats: quality".format(rec_update_counter)) - - # Calculate quality_offset of devices - dev_sq = ( - session.query( - RelationStats.device_id, (func.sum(RelationStats.beacon_count * (RelationStats.quality - ReceiverStats.quality)) / (func.sum(RelationStats.beacon_count))).label("quality_offset") - ) - .filter(RelationStats.date == date) - .filter(and_(RelationStats.receiver_id == ReceiverStats.receiver_id, RelationStats.date == ReceiverStats.date)) - .group_by(RelationStats.device_id) - .subquery() - ) - - dev_upd = update(DeviceStats).where(and_(DeviceStats.date == date, DeviceStats.device_id == dev_sq.c.device_id)).values({"quality_offset": dev_sq.c.quality_offset}) - - dev_result = session.execute(dev_upd) - dev_update_counter = dev_result.rowcount - session.commit() - logger.warn("Updated {} DeviceStats: quality_offset".format(dev_update_counter)) - - # Calculate quality_offset of receivers - rec_sq = ( - session.query( - RelationStats.receiver_id, (func.sum(RelationStats.beacon_count * (RelationStats.quality - DeviceStats.quality)) / (func.sum(RelationStats.beacon_count))).label("quality_offset") - ) - .filter(RelationStats.date == date) - .filter(and_(RelationStats.device_id == DeviceStats.device_id, RelationStats.date == DeviceStats.date)) - .group_by(RelationStats.receiver_id) - .subquery() - ) - - rec_upd = update(ReceiverStats).where(and_(ReceiverStats.date == date, ReceiverStats.receiver_id == rec_sq.c.receiver_id)).values({"quality_offset": rec_sq.c.quality_offset}) - - rec_result = session.execute(rec_upd) - rec_update_counter = rec_result.rowcount - session.commit() - logger.warn("Updated {} ReceiverStats: quality_offset".format(rec_update_counter)) - - return "Updated {} DeviceStats and {} ReceiverStats".format(dev_update_counter, rec_update_counter) - - -def update_receivers(session, logger=None): - """Update receivers with stats.""" - - if logger is None: - logger = current_app.logger - - receiver_stats = ( - session.query( - distinct(ReceiverStats.receiver_id).label("receiver_id"), - func.first_value(ReceiverStats.firstseen) - .over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.firstseen == null(), None)], else_=ReceiverStats.date).asc().nullslast()) - .label("firstseen"), - func.first_value(ReceiverStats.lastseen) - .over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.lastseen == null(), None)], else_=ReceiverStats.date).desc().nullslast()) - .label("lastseen"), - func.first_value(ReceiverStats.location_wkt) - .over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.location_wkt == null(), None)], else_=ReceiverStats.date).desc().nullslast()) - .label("location_wkt"), - func.first_value(ReceiverStats.altitude) - .over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.altitude == null(), None)], else_=ReceiverStats.date).desc().nullslast()) - .label("altitude"), - func.first_value(ReceiverStats.version) - .over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.version == null(), None)], else_=ReceiverStats.date).desc().nullslast()) - .label("version"), - func.first_value(ReceiverStats.platform) - .over(partition_by=ReceiverStats.receiver_id, order_by=case([(ReceiverStats.platform == null(), None)], else_=ReceiverStats.date).desc().nullslast()) - .label("platform"), - ) - .order_by(ReceiverStats.receiver_id) - .subquery() - ) - - upd = ( - update(Receiver) - .where(and_(Receiver.id == receiver_stats.c.receiver_id)) - .values( - { - "firstseen": receiver_stats.c.firstseen, - "lastseen": receiver_stats.c.lastseen, - "location": receiver_stats.c.location_wkt, - "altitude": receiver_stats.c.altitude, - "version": receiver_stats.c.version, - "platform": receiver_stats.c.platform, - } - ) - ) - - result = session.execute(upd) - update_counter = result.rowcount - session.commit() - logger.warn("Updated {} Receivers".format(update_counter)) - - return "Updated {} Receivers".format(update_counter) - - -def update_devices(session, logger=None): - """Update devices with stats.""" - - if logger is None: - logger = current_app.logger - - device_stats = ( - session.query( - distinct(DeviceStats.device_id).label("device_id"), - func.first_value(DeviceStats.name).over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.name == null(), None)], else_=DeviceStats.date).desc().nullslast()).label("name"), - func.first_value(DeviceStats.firstseen) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.firstseen == null(), None)], else_=DeviceStats.date).asc().nullslast()) - .label("firstseen"), - func.max(DeviceStats.lastseen) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.lastseen == null(), None)], else_=DeviceStats.date).desc().nullslast()) - .label("lastseen"), - func.first_value(DeviceStats.aircraft_type) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.aircraft_type == null(), None)], else_=DeviceStats.date).desc().nullslast()) - .label("aircraft_type"), - func.first_value(DeviceStats.stealth) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.stealth == null(), None)], else_=DeviceStats.date).desc().nullslast()) - .label("stealth"), - func.first_value(DeviceStats.software_version) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.software_version == null(), None)], else_=DeviceStats.date).desc().nullslast()) - .label("software_version"), - func.first_value(DeviceStats.hardware_version) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.hardware_version == null(), None)], else_=DeviceStats.date).desc().nullslast()) - .label("hardware_version"), - func.first_value(DeviceStats.real_address) - .over(partition_by=DeviceStats.device_id, order_by=case([(DeviceStats.real_address == null(), None)], else_=DeviceStats.date).desc().nullslast()) - .label("real_address"), - ) - .order_by(DeviceStats.device_id) - .subquery() - ) - - upd = ( - update(Device) - .where(and_(Device.id == device_stats.c.device_id)) - .values( - { - "name": device_stats.c.name, - "firstseen": device_stats.c.firstseen, - "lastseen": device_stats.c.lastseen, - "aircraft_type": device_stats.c.aircraft_type, - "stealth": device_stats.c.stealth, - "software_version": device_stats.c.software_version, - "hardware_version": device_stats.c.hardware_version, - "real_address": device_stats.c.real_address, - } - ) - ) - - result = session.execute(upd) - update_counter = result.rowcount - session.commit() - logger.warn("Updated {} Devices".format(update_counter)) - - return "Updated {} Devices".format(update_counter) diff --git a/app/collect/takeoff_landings.py b/app/collect/takeoff_landings.py deleted file mode 100644 index d60486d..0000000 --- a/app/collect/takeoff_landings.py +++ /dev/null @@ -1,145 +0,0 @@ -from datetime import timedelta - -from flask import current_app -from sqlalchemy import and_, or_, insert, between, exists -from sqlalchemy.sql import func, null -from sqlalchemy.sql.expression import case - -from app.model import AircraftBeacon, Device, TakeoffLanding, Airport - - -def update_entries(session, start, end, logger=None): - """Compute takeoffs and landings.""" - - if logger is None: - logger = current_app.logger - - logger.info("Compute takeoffs and landings.") - - # considered time interval should not exceed a complete day - if end - start > timedelta(days=1): - abort_message = "TakeoffLanding: timeinterval start='{}' and end='{}' is too big.".format(start, end) - logger.warn(abort_message) - return abort_message - - # check if we have any airport - airports_query = session.query(Airport).limit(1) - if not airports_query.all(): - abort_message = "TakeoffLanding: Cannot calculate takeoff and landings without any airport! Please import airports first." - logger.warn(abort_message) - return abort_message - - # takeoff / landing detection is based on 3 consecutive points all below a certain altitude AGL - takeoff_speed = 55 # takeoff detection: 1st point below, 2nd and 3rd above this limit - landing_speed = 40 # landing detection: 1st point above, 2nd and 3rd below this limit - min_takeoff_climb_rate = -5 # takeoff detection: glider should not sink too much - max_landing_climb_rate = 5 # landing detection: glider should not climb too much - duration = 100 # the points must not exceed this duration - radius = 5000 # the points must not exceed this radius around the 2nd point - max_agl = 200 # takeoff / landing must not exceed this altitude AGL - - # get beacons for selected time range, one per address and timestamp - sq = ( - session.query(AircraftBeacon) - .distinct(AircraftBeacon.address, AircraftBeacon.timestamp) - .order_by(AircraftBeacon.address, AircraftBeacon.timestamp, AircraftBeacon.error_count) - .filter(AircraftBeacon.agl < max_agl) - .filter(between(AircraftBeacon.timestamp, start, end)) - .subquery() - ) - - # make a query with current, previous and next position - sq2 = session.query( - sq.c.address, - func.lag(sq.c.address).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("address_prev"), - func.lead(sq.c.address).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("address_next"), - sq.c.timestamp, - func.lag(sq.c.timestamp).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("timestamp_prev"), - func.lead(sq.c.timestamp).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("timestamp_next"), - sq.c.location, - func.lag(sq.c.location).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("location_wkt_prev"), - func.lead(sq.c.location).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("location_wkt_next"), - sq.c.track, - func.lag(sq.c.track).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("track_prev"), - func.lead(sq.c.track).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("track_next"), - sq.c.ground_speed, - func.lag(sq.c.ground_speed).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("ground_speed_prev"), - func.lead(sq.c.ground_speed).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("ground_speed_next"), - sq.c.altitude, - func.lag(sq.c.altitude).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("altitude_prev"), - func.lead(sq.c.altitude).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("altitude_next"), - sq.c.climb_rate, - func.lag(sq.c.climb_rate).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("climb_rate_prev"), - func.lead(sq.c.climb_rate).over(partition_by=sq.c.address, order_by=sq.c.timestamp).label("climb_rate_next"), - ).subquery() - - # consider only positions with predecessor and successor and limit distance and duration between points - sq3 = ( - session.query(sq2) - .filter(and_(sq2.c.address_prev != null(), sq2.c.address_next != null())) - .filter(and_(func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_prev) < radius, func.ST_DistanceSphere(sq2.c.location, sq2.c.location_wkt_next) < radius)) - .filter(sq2.c.timestamp_next - sq2.c.timestamp_prev < timedelta(seconds=duration)) - .subquery() - ) - - # find possible takeoffs and landings - sq4 = ( - session.query( - sq3.c.timestamp, - case( - [ - (sq3.c.ground_speed > takeoff_speed, sq3.c.location_wkt_prev), # on takeoff we take the location from the previous fix because it is nearer to the airport - (sq3.c.ground_speed <= takeoff_speed, sq3.c.location), - ] - ).label("location"), - case([(sq3.c.ground_speed > landing_speed, sq3.c.track), (sq3.c.ground_speed <= landing_speed, sq3.c.track_prev)]).label( - "track" - ), # on landing we take the track from the previous fix because gliders tend to leave the runway quickly - sq3.c.ground_speed, - sq3.c.altitude, - case([(sq3.c.ground_speed > takeoff_speed, True), (sq3.c.ground_speed < landing_speed, False)]).label("is_takeoff"), - sq3.c.address, - ) - .filter( - or_( - and_(sq3.c.ground_speed_prev < takeoff_speed, sq3.c.ground_speed > takeoff_speed, sq3.c.ground_speed_next > takeoff_speed, sq3.c.climb_rate > min_takeoff_climb_rate), # takeoff - and_(sq3.c.ground_speed_prev > landing_speed, sq3.c.ground_speed < landing_speed, sq3.c.ground_speed_next < landing_speed, sq3.c.climb_rate < max_landing_climb_rate), # landing - ) - ) - .subquery() - ) - - # get the device id instead of the address and consider them if the are near airports ... - sq5 = ( - session.query( - sq4.c.timestamp, sq4.c.track, sq4.c.is_takeoff, Device.id.label("device_id"), Airport.id.label("airport_id"), func.ST_DistanceSphere(sq4.c.location, Airport.location_wkt).label("airport_distance") - ) - .filter(and_(sq4.c.address == Device.address, - func.ST_Within(sq4.c.location, Airport.border), - between(Airport.style, 2, 5))) - .subquery() - ) - - # ... and take the nearest airport - sq6 = ( - session.query(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.device_id, sq5.c.airport_id) - .distinct(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.device_id) - .order_by(sq5.c.timestamp, sq5.c.track, sq5.c.is_takeoff, sq5.c.device_id, sq5.c.airport_distance) - .subquery() - ) - - # consider them only if they are not already existing in db - takeoff_landing_query = session.query(sq6).filter( - ~exists().where(and_(TakeoffLanding.timestamp == sq6.c.timestamp, TakeoffLanding.device_id == sq6.c.device_id, TakeoffLanding.airport_id == sq6.c.airport_id)) - ) - - # ... and save them - ins = insert(TakeoffLanding).from_select((TakeoffLanding.timestamp, TakeoffLanding.track, TakeoffLanding.is_takeoff, TakeoffLanding.device_id, TakeoffLanding.airport_id), takeoff_landing_query) - - result = session.execute(ins) - session.commit() - insert_counter = result.rowcount - - finish_message = "TakeoffLandings: {} inserted".format(insert_counter) - logger.info(finish_message) - return finish_message diff --git a/app/collect/timescaledb_views.py b/app/collect/timescaledb_views.py new file mode 100644 index 0000000..b577b04 --- /dev/null +++ b/app/collect/timescaledb_views.py @@ -0,0 +1,167 @@ +from app import db +from app.utils import get_sql_trustworthy + +SQL_TRUSTWORTHY = get_sql_trustworthy(source_table_alias='sp') + + +def create_views(): + db.session.execute(""" + DROP VIEW IF EXISTS receiver_ranking CASCADE; + + CREATE VIEW receiver_ranking AS + SELECT + r.name AS receiver_name, + r.id AS receiver_id, + MAX(rs.max_distance) AS max_distance, + SUM(rs.max_normalized_quality * rs.messages_count) / SUM(rs.messages_count) AS max_normalized_quality, + SUM(rs.messages_count) AS messages_count, + COUNT(DISTINCT rs.sender_id) AS senders_count, + COUNT(DISTINCT rs.location_mgrs_short) AS coverage_count + FROM coverage_statistics AS rs + INNER JOIN receivers AS r ON rs.receiver_id = r.id + WHERE rs.date = NOW()::date AND rs.is_trustworthy IS TRUE AND rs.max_distance IS NOT NULL + GROUP BY rs.date, r.name, r.id + ORDER BY max_distance DESC; + """) + + db.session.execute(""" + DROP VIEW IF EXISTS sender_ranking CASCADE; + + CREATE VIEW sender_ranking AS + SELECT + s.name, + s.id AS sender_id, + MAX(rs.max_distance) AS max_distance, + SUM(rs.max_normalized_quality * rs.messages_count) / SUM(rs.messages_count) AS max_normalized_quality, + SUM(rs.messages_count) AS messages_count, + COUNT(DISTINCT rs.receiver_id) AS receivers_count, + COUNT(DISTINCT rs.location_mgrs_short) AS coverage_count + FROM coverage_statistics AS rs + INNER JOIN senders AS s ON rs.sender_id = s.id + WHERE rs.date = NOW()::date AND rs.is_trustworthy IS TRUE AND rs.max_distance IS NOT NULL + GROUP BY rs.date, s.name, s.id + ORDER BY max_distance DESC; + """) + + db.session.commit() + + +def create_timescaledb_views(): + # 1. Since the reference_timestamps are strictly increasing we can set + # the parameter 'refresh_lag' to a very short time so the materialization + # starts right after the bucket is finished + # 2. The feature realtime aggregation from TimescaleDB is quite time consuming. + # So we set materialized_only=true + + # --- Sender statistics --- + # These stats will be used in the daily ranking, so we make the bucket < 1d + db.session.execute(f""" + DROP VIEW IF EXISTS sender_stats_1h CASCADE; + + CREATE VIEW sender_stats_1h + WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='5 minutes') AS + SELECT + time_bucket(INTERVAL '1 hour', sp.reference_timestamp) AS bucket, + sp.name, + ({SQL_TRUSTWORTHY}) AS is_trustworthy, + COUNT(sp.*) AS beacon_count, + MAX(sp.distance) AS max_distance, + MIN(sp.altitude) AS min_altitude, + MAX(sp.altitude) AS max_altitude + + FROM sender_positions AS sp + GROUP BY bucket, sp.name, is_trustworthy; + """) + + # ... and just for curiosity also bucket = 1d + db.session.execute(f""" + DROP VIEW IF EXISTS sender_stats_1d CASCADE; + + CREATE VIEW sender_stats_1d + WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='1 hour') AS + SELECT + time_bucket(INTERVAL '1 day', sp.reference_timestamp) AS bucket, + sp.name, + ({SQL_TRUSTWORTHY}) AS is_trustworthy, + COUNT(sp.*) AS beacon_count, + MAX(sp.distance) AS max_distance, + MIN(sp.altitude) AS min_altitude, + MAX(sp.altitude) AS max_altitude + + FROM sender_positions AS sp + GROUP BY bucket, sp.name, is_trustworthy; + """) + + # --- Receiver statistics --- + # These stats will be used in the daily ranking, so we make the bucket < 1d + db.session.execute(f""" + DROP VIEW IF EXISTS receiver_stats_1h CASCADE; + + CREATE VIEW receiver_stats_1h + WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='5 minutes') AS + SELECT + time_bucket(INTERVAL '1 hour', sp.reference_timestamp) AS bucket, + sp.receiver_name, + ({SQL_TRUSTWORTHY}) AS is_trustworthy, + COUNT(sp.*) AS beacon_count, + MAX(sp.distance) AS max_distance, + MIN(sp.altitude) AS min_altitude, + MAX(sp.altitude) AS max_altitude + + FROM sender_positions AS sp + GROUP BY bucket, sp.receiver_name, is_trustworthy; + """) + + # ... and just for curiosity also bucket = 1d + db.session.execute(f""" + DROP VIEW IF EXISTS receiver_stats_1d CASCADE; + + CREATE VIEW receiver_stats_1d + WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='1 hour') AS + SELECT + time_bucket(INTERVAL '1 day', sp.reference_timestamp) AS bucket, + sp.receiver_name, + ({SQL_TRUSTWORTHY}) AS is_trustworthy, + COUNT(sp.*) AS beacon_count, + MAX(sp.distance) AS max_distance, + MIN(sp.altitude) AS min_altitude, + MAX(sp.altitude) AS max_altitude + + FROM sender_positions AS sp + GROUP BY bucket, sp.receiver_name, is_trustworthy; + """) + + # --- Relation statistics (sender <-> receiver) --- + # these stats will be used on a >= 1d basis, so we make the bucket = 1d + db.session.execute(f""" + DROP VIEW IF EXISTS relation_stats_1d CASCADE; + + CREATE VIEW relation_stats_1d + WITH (timescaledb.continuous, timescaledb.materialized_only=true, timescaledb.refresh_lag='1 hour') AS + SELECT + time_bucket(INTERVAL '1 day', sp.reference_timestamp) AS bucket, + sp.name, + sp.receiver_name, + ({SQL_TRUSTWORTHY}) AS is_trustworthy, + COUNT(sp.*) AS beacon_count, + MAX(sp.normalized_quality) AS max_normalized_quality, + MAX(sp.distance) AS max_distance + + FROM sender_positions AS sp + GROUP BY bucket, sp.name, sp.receiver_name, is_trustworthy; + """) + + db.session.commit() + + +""" +class MyView(db.Model): + __table__ = db.Table( + 'device_stats', db.metadata, + db.Column('bucket', db.DateTime, primary_key=True), + db.Column('name', db.String, primary_key=True), + db.Column('beacon_count', db.Integer), + autoload=True, + autoload_with=db.engine + ) +""" diff --git a/app/commands/__init__.py b/app/commands/__init__.py index 3e442f0..e8f43a4 100644 --- a/app/commands/__init__.py +++ b/app/commands/__init__.py @@ -3,7 +3,6 @@ from .export import user_cli as export_cli from .flights import user_cli as flights_cli from .gateway import user_cli as gateway_cli from .logbook import user_cli as logbook_cli -from .stats import user_cli as stats_cli def register(app): @@ -12,4 +11,3 @@ def register(app): app.cli.add_command(flights_cli) app.cli.add_command(gateway_cli) app.cli.add_command(logbook_cli) - app.cli.add_command(stats_cli) diff --git a/app/commands/database.py b/app/commands/database.py index 9f1c550..2901364 100644 --- a/app/commands/database.py +++ b/app/commands/database.py @@ -5,9 +5,10 @@ import click from datetime import datetime from sqlalchemy.sql import func -from app.collect.database import update_device_infos, update_country_code -from app.model import AircraftBeacon, DeviceInfoOrigin +from app.collect.database import update_device_infos +from app.model import SenderPosition, SenderInfoOrigin from app.utils import get_airports, get_days +from app.collect.timescaledb_views import create_timescaledb_views, create_views from app import db @@ -22,7 +23,7 @@ def get_database_days(start, end): """Returns the first and the last day in aircraft_beacons table.""" if start is None and end is None: - days_from_db = db.session.query(func.min(AircraftBeacon.timestamp).label("first_day"), func.max(AircraftBeacon.timestamp).label("last_day")).one() + days_from_db = db.session.query(func.min(SenderPosition.timestamp).label("first_day"), func.max(SenderPosition.timestamp).label("last_day")).one() start = days_from_db[0].date() end = days_from_db[1].date() else: @@ -42,29 +43,26 @@ def info(): @user_cli.command("init") def init(): - """Initialize the database.""" + """Initialize the database (with PostGIS and TimescaleDB extensions).""" from alembic.config import Config from alembic import command + # Create PostGIS and PostGIS extensions db.session.execute("CREATE EXTENSION IF NOT EXISTS postgis;") db.session.execute("CREATE EXTENSION IF NOT EXISTS btree_gist;") + db.session.execute("CREATE EXTENSION IF NOT EXISTS timescaledb;") db.session.commit() + + # Create Scheme db.create_all() - print("Done.") - - -@user_cli.command("init_timescaledb") -def init_timescaledb(): - """Initialize TimescaleDB features.""" - - db.session.execute("CREATE EXTENSION IF NOT EXISTS timescaledb;") - db.session.execute("SELECT create_hypertable('aircraft_beacons', 'timestamp', chunk_time_interval => interval '6 hours', if_not_exists => TRUE);") - db.session.execute("SELECT create_hypertable('receiver_beacons', 'timestamp', chunk_time_interval => interval '6 hours', if_not_exists => TRUE);") + # Change (sender|receiver)_positions to TimescaleDB table + db.session.execute("SELECT create_hypertable('sender_positions', 'reference_timestamp', chunk_time_interval => interval '3 hours', if_not_exists => TRUE);") + db.session.execute("SELECT create_hypertable('receiver_positions', 'reference_timestamp', chunk_time_interval => interval '1 day', if_not_exists => TRUE);") db.session.commit() - print("Done.") + print("Initialized the database (with PostGIS and TimescaleDB extensions).") @user_cli.command("drop") @@ -83,7 +81,7 @@ def import_ddb(): """Import registered devices from the DDB.""" print("Import registered devices fom the DDB...") - counter = update_device_infos(db.session, DeviceInfoOrigin.OGN_DDB) + counter = update_device_infos(SenderInfoOrigin.OGN_DDB) print("Imported %i devices." % counter) @@ -93,7 +91,7 @@ def import_file(path="tests/custom_ddb.txt"): """Import registered devices from a local file.""" print("Import registered devices from '{}'...".format(path)) - counter = update_device_infos(db.session, DeviceInfoOrigin.USER_DEFINED, path=path) + counter = update_device_infos(SenderInfoOrigin.USER_DEFINED, path=path) print("Imported %i devices." % counter) @@ -103,7 +101,7 @@ def import_flarmnet(path=None): """Import registered devices from a local file.""" print("Import registered devices from '{}'...".format("internet" if path is None else path)) - counter = update_device_infos(db.session, DeviceInfoOrigin.FLARMNET, path=path) + counter = update_device_infos(SenderInfoOrigin.FLARMNET, path=path) print("Imported %i devices." % counter) @@ -116,13 +114,23 @@ def import_airports(path="tests/SeeYou.cup"): airports = get_airports(path) db.session.bulk_save_objects(airports) db.session.commit() - db.session.execute("UPDATE airports SET border = ST_Expand(location, 0.05)") + # TODO: SRID 4087 ist nicht korrekt, aber spherical mercator 3857 wirft hier Fehler + db.session.execute("UPDATE airports AS a SET border = ST_Transform(ST_Buffer(ST_Transform(location, 4087), 1.5 * GREATEST(500, a.runway_length)), 4326);") db.session.commit() print("Imported {} airports.".format(len(airports))) -@user_cli.command("update_country_codes") -def update_country_codes(): - """Update country codes of all receivers.""" +@user_cli.command("create_timescaledb_views") +def cmd_create_timescaledb_views(): + """Create TimescaleDB views.""" - update_country_code(session=db.session) + create_timescaledb_views() + print("Done") + + +@user_cli.command("create_views") +def cmd_create_views(): + """Create views.""" + + create_views() + print("Done") diff --git a/app/commands/export.py b/app/commands/export.py index 11f0756..a6f07c4 100644 --- a/app/commands/export.py +++ b/app/commands/export.py @@ -4,15 +4,89 @@ import click import datetime import re import csv +import os + +from sqlalchemy.orm.exc import NoResultFound from aerofiles.igc import Writer -from app.model import AircraftBeacon, Device +from app.model import SenderPosition, Sender from app import db user_cli = AppGroup("export") user_cli.help = "Export data in several file formats." +@user_cli.command("debug_sql") +@click.argument("start") +@click.argument("end") +@click.argument("name") +def debug_sql(start, end, name): + """Export data (sender_positions and receivers) as sql for debugging (and/or creating test cases).""" + + # First: get all the positions (and the receiver names for later) + sql_sender_positions = f""" + SELECT reference_timestamp, name, receiver_name, timestamp, location, track, ground_speed, altitude, aircraft_type, climb_rate, turn_rate, distance, bearing, agl + FROM sender_positions + WHERE reference_timestamp BETWEEN '{start}' AND '{end}' AND name = '{name}' + ORDER BY reference_timestamp; + """ + + receiver_names = [] + sender_position_values = [] + results = db.session.execute(sql_sender_positions) + for row in results: + if row[2] not in receiver_names: + receiver_names.append("'" + row[2] + "'") + row = [f"'{r}'" if r else "DEFAULT" for r in row] + sender_position_values.append(f"({','.join(row)})") + + # Second: get the receivers + sql_receivers = f""" + SELECT name, location + FROM receivers + WHERE name IN ({','.join(receiver_names)}); + """ + + receiver_values = [] + results = db.session.execute(sql_receivers) + for row in results: + row = [f"'{r}'" if r else "DEFAULT" for r in row] + receiver_values.append(f"({','.join(row)})") + + # Third: get the airports + sql_airports = f""" + SELECT DISTINCT a.name, a.location, a.altitude, a.style, a.border + FROM airports AS a, receivers AS r + WHERE + r.name IN ({','.join(receiver_names)}) + AND ST_Within(r.location, ST_Buffer(a.location, 0.2)) + AND a.style IN (2,4,5); + """ + + airport_values = [] + results = db.session.execute(sql_airports) + for row in results: + row = [f"'{r}'" if r else "DEFAULT" for r in row] + airport_values.append(f"({','.join(row)})") + + # Last: write all into file + with open(f'{start}_{end}_{name}.sql', 'w') as file: + file.write('/*\n') + file.write('OGN Python SQL Export\n') + file.write(f'Created by: {os.getlogin()}\n') + file.write(f'Created at: {datetime.datetime.utcnow()}\n') + file.write('*/\n\n') + + file.write("INSERT INTO airports(name, location, altitude, style, border) VALUES\n") + file.write(',\n'.join(airport_values) + ';\n\n') + + file.write("INSERT INTO receivers(name, location) VALUES\n") + file.write(',\n'.join(receiver_values) + ';\n\n') + + file.write("INSERT INTO sender_positions(reference_timestamp, name, receiver_name, timestamp, location, track, ground_speed, altitude, aircraft_type, climb_rate, turn_rate, distance, bearing, agl) VALUES\n") + file.write(',\n'.join(sender_position_values) + ';\n\n') + + @user_cli.command("cup") def cup(): """Export receiver waypoints as '.cup'.""" @@ -60,18 +134,18 @@ def cup(): @click.argument("date") def igc(address, date): """Export igc file for
at .""" - if not re.match(".{6}", address): - print("Address {} not valid.".format(address)) + if not re.match("[0-9A-F]{6}", address): + print(f"Address '{address}' not valid.") + return + + try: + sender = db.session.query(Sender).filter(Sender.address == address).one() + except NoResultFound as e: + print(f"No data for '{address}' in the DB") return if not re.match(r"\d{4}-\d{2}-\d{2}", date): - print("Date {} not valid.".format(date)) - return - - device_id = db.session.query(Device.id).filter(Device.address == address).first() - - if device_id is None: - print("Device with address '{}' not found.".format(address)) + print(f"Date {date} not valid.") return with open("sample.igc", "wb") as fp: @@ -83,27 +157,26 @@ def igc(address, date): "logger_id": "OGN", "date": datetime.date(1987, 2, 24), "fix_accuracy": 50, - "pilot": "Konstantin Gruendger", + "pilot": "Unknown", "copilot": "", - "glider_type": "Duo Discus", - "glider_id": "D-KKHH", - "firmware_version": "2.2", - "hardware_version": "2", - "logger_type": "LXNAVIGATION,LX8000F", - "gps_receiver": "uBLOX LEA-4S-2,16,max9000m", - "pressure_sensor": "INTERSEMA,MS5534A,max10000m", - "competition_id": "2H", - "competition_class": "Doubleseater", + "glider_type": sender.infos[0].aircraft if len(sender.infos) > 0 else '', + "glider_id": sender.infos[0].registration if len(sender.infos) > 0 else '', + "firmware_version": sender.software_version, + "hardware_version": sender.hardware_version, + "logger_type": "OGN", + "gps_receiver": "unknown", + "pressure_sensor": "unknown", + "competition_id": sender.infos[0].competition if len(sender.infos) > 0 else '', + "competition_class": "unknown", } ) points = ( - db.session.query(AircraftBeacon) - .filter(AircraftBeacon.device_id == device_id) - .filter(AircraftBeacon.timestamp > date + " 00:00:00") - .filter(AircraftBeacon.timestamp < date + " 23:59:59") - .order_by(AircraftBeacon.timestamp) + db.session.query(SenderPosition) + .filter(db.between(SenderPosition.reference_timestamp, f"{date} 00:00:00", f"{date} 23:59:59")) + .filter(SenderPosition.name == sender.name) + .order_by(SenderPosition.timestamp) ) - for point in points.all(): + for point in points: writer.write_fix(point.timestamp.time(), latitude=point.location.latitude, longitude=point.location.longitude, valid=True, pressure_alt=point.altitude, gps_alt=point.altitude) diff --git a/app/commands/flights.py b/app/commands/flights.py index 5c81f63..035546c 100644 --- a/app/commands/flights.py +++ b/app/commands/flights.py @@ -6,119 +6,11 @@ from tqdm import tqdm from app.commands.database import get_database_days from app import db +from app.collect.flights import compute_flights, compute_gaps user_cli = AppGroup("flights") user_cli.help = "Create 2D flight paths from data." -NOTHING = "" -CONTEST_RELEVANT = "AND agl < 1000" -LOW_PASS = "AND agl < 50 and ground_speed > 250" - - -def compute_gaps(session, date): - query = """ - INSERT INTO flights2d(date, flight_type, device_id, path) - SELECT '{date}' AS date, - 3 AS flight_type, - sq3.device_id, - ST_Collect(sq3.path) - FROM ( - SELECT sq2.d1 device_id, - ST_MakeLine(sq2.l1, sq2.l2) path - FROM - ( - SELECT sq.timestamp t1, - LAG(sq.timestamp) OVER ( PARTITION BY sq.timestamp::DATE, sq.device_id ORDER BY sq.timestamp) t2, - sq.location l1, - LAG(sq.location) OVER ( PARTITION BY sq.timestamp::DATE, sq.device_id ORDER BY sq.timestamp) l2, - sq.device_id d1, - LAG(sq.device_id) OVER ( PARTITION BY sq.timestamp::DATE, sq.device_id ORDER BY sq.timestamp) d2 - FROM - ( - SELECT DISTINCT ON (device_id, timestamp) timestamp, device_id, location, agl - FROM aircraft_beacons - WHERE timestamp BETWEEN '{date} 00:00:00' AND '{date} 23:59:59' AND agl > 300 - ORDER BY device_id, timestamp, error_count - ) sq - ) sq2 - WHERE EXTRACT(epoch FROM sq2.t1 - sq2.t2) > 300 - AND ST_DistanceSphere(sq2.l1, sq2.l2) / EXTRACT(epoch FROM sq2.t1 - sq2.t2) BETWEEN 15 AND 50 - ) sq3 - GROUP BY sq3.device_id - ON CONFLICT DO NOTHING; - """.format( - date=date.strftime("%Y-%m-%d") - ) - - session.execute(query) - session.commit() - - -def compute_flights2d(session, date, flight_type): - if flight_type == 0: - filter = NOTHING - elif flight_type == 1: - filter = CONTEST_RELEVANT - elif flight_type == 2: - filter = LOW_PASS - - query = """ - INSERT INTO flights2d - ( - date, - flight_type, - device_id, - path, - path_simple - ) - SELECT '{date}' AS date, - {flight_type} as flight_type, - sq5.device_id, - st_collect(sq5.linestring order BY sq5.part) multilinestring, - st_collect(st_simplify(sq5.linestring, 0.0001) ORDER BY sq5.part) simple_multilinestring - FROM ( - SELECT sq4.device_id, - sq4.part, - st_makeline(sq4.location ORDER BY sq4.timestamp) linestring - FROM ( - SELECT sq3.timestamp, - sq3.location, - sq3.device_id, - sum(sq3.ping) OVER (partition BY sq3.device_id ORDER BY sq3.timestamp) part - FROM ( - SELECT sq2.t1 AS timestamp, - sq2.l1 AS location, - sq2.d1 device_id, - CASE - WHEN sq2.t1 - sq2.t2 < interval'100s' AND ST_DistanceSphere(sq2.l1, sq2.l2) < 1000 THEN 0 - ELSE 1 - END AS ping - FROM ( - SELECT sq.timestamp t1, - lag(sq.timestamp) OVER (partition BY sq.device_id ORDER BY sq.timestamp) t2, - sq.location l1, - lag(sq.location) OVER (partition BY sq.device_id ORDER BY sq.timestamp) l2, - sq.device_id d1, - lag(sq.device_id) OVER (partition BY sq.device_id ORDER BY sq.timestamp) d2 - FROM ( - SELECT DISTINCT ON (device_id, timestamp) timestamp, device_id, location - FROM aircraft_beacons - WHERE timestamp BETWEEN '{date} 00:00:00' AND '{date} 23:59:59' {filter} - ORDER BY device_id, timestamp, error_count - ) sq - ) sq2 - ) sq3 - ) sq4 - GROUP BY sq4.device_id, sq4.part - ) sq5 - GROUP BY sq5.device_id - ON CONFLICT DO NOTHING; - """.format( - date=date.strftime("%Y-%m-%d"), flight_type=flight_type, filter=filter - ) - session.execute(query) - session.commit() - @user_cli.command("create") @click.argument("start") @@ -133,6 +25,6 @@ def create(start, end, flight_type): for single_date in pbar: pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d")) if flight_type <= 2: - result = compute_flights2d(session=db.session, date=single_date, flight_type=flight_type) + result = compute_flights(date=single_date, flight_type=flight_type) else: - result = compute_gaps(session=db.session, date=single_date) + result = compute_gaps(date=single_date) diff --git a/app/commands/gateway.py b/app/commands/gateway.py index 7691e0a..d22cf87 100644 --- a/app/commands/gateway.py +++ b/app/commands/gateway.py @@ -1,49 +1,103 @@ import os -import datetime +from datetime import datetime, timezone +import time from flask import current_app from flask.cli import AppGroup import click +from tqdm import tqdm from ogn.client import AprsClient -from app.gateway.bulkimport import convert, DbFeeder +from app import redis_client +from app.gateway.beacon_conversion import aprs_string_to_message +from app.gateway.message_handling import receiver_status_message_to_csv_string, receiver_position_message_to_csv_string, sender_position_message_to_csv_string +from app.collect.gateway import transfer_from_redis_to_database user_cli = AppGroup("gateway") user_cli.help = "Connection to APRS servers." @user_cli.command("run") -def run(aprs_user="anon-dev"): - """Run the aprs client and feed the DB with incoming data.""" +@click.option("--aprs_filter", default='') +def run(aprs_filter): + """ + Run the aprs client, parse the incoming data and put it to redis. + """ - # User input validation - if len(aprs_user) < 3 or len(aprs_user) > 9: - print("aprs_user must be a string of 3-9 characters.") - return + import logging + logging.basicConfig(level=logging.INFO, format='%(asctime)s %(name)-17s %(levelname)-8s %(message)s') current_app.logger.warning("Start ogn gateway") - client = AprsClient(aprs_user) + client = AprsClient(current_app.config['APRS_USER'], aprs_filter) client.connect() - with DbFeeder(prefix='continuous_import', reference_timestamp=datetime.utcnow, reference_timestamp_autoupdate=True) as feeder: - try: - client.run(callback=lambda x: feeder.add(x), autoreconnect=True) - except KeyboardInterrupt: - current_app.logger.warning("\nStop ogn gateway") + def insert_into_redis(aprs_string): + # Convert aprs_string to message dict, add MGRS Position, flatten gps precision, etc. etc. ... + message = aprs_string_to_message(aprs_string) + if message is None: + return + + # separate between tables (receiver/sender) and aprs_type (status/position) + if message['beacon_type'] in ('aprs_receiver', 'receiver'): + if message['aprs_type'] == 'status': + redis_target = 'receiver_status' + csv_string = receiver_status_message_to_csv_string(message, none_character=r'\N') + elif message['aprs_type'] == 'position': + redis_target = 'receiver_position' + csv_string = receiver_position_message_to_csv_string(message, none_character=r'\N') + else: + return + else: + if message['aprs_type'] == 'status': + return # no interesting data we want to keep + elif message['aprs_type'] == 'position': + redis_target = 'sender_position' + csv_string = sender_position_message_to_csv_string(message, none_character=r'\N') + else: + return + + mapping = {csv_string: str(time.time())} + + redis_client.zadd(name=redis_target, mapping=mapping, nx=True) + insert_into_redis.beacon_counter += 1 + + current_minute = datetime.utcnow().minute + if current_minute != insert_into_redis.last_minute: + current_app.logger.info(f"{insert_into_redis.beacon_counter:7d}/min") + insert_into_redis.beacon_counter = 0 + insert_into_redis.last_minute = current_minute + + insert_into_redis.beacon_counter = 0 + insert_into_redis.last_minute = datetime.utcnow().minute + + try: + client.run(callback=insert_into_redis, autoreconnect=True) + except KeyboardInterrupt: + current_app.logger.warning("\nStop ogn gateway") client.disconnect() -@user_cli.command("convert") -@click.argument("path") -def file_import(path): - """Convert APRS logfiles into csv files for fast bulk import.""" +@user_cli.command("transfer") +def transfer(): + """Transfer data from redis to the database.""" - logfiles = [] - for (root, dirs, files) in os.walk(path): - for file in sorted(files): - logfiles.append(os.path.join(root, file)) + transfer_from_redis_to_database() - for logfile in logfiles: - convert(logfile) + +@user_cli.command("printout") +@click.option("--aprs_filter", default='') +def printout(aprs_filter): + """Run the aprs client and just print out the data stream.""" + + current_app.logger.warning("Start ogn gateway") + client = AprsClient(current_app.config['APRS_USER'], aprs_filter=aprs_filter) + client.connect() + + try: + client.run(callback=lambda x: print(f"{datetime.utcnow()}: {x}"), autoreconnect=True) + except KeyboardInterrupt: + current_app.logger.warning("\nStop ogn gateway") + + client.disconnect() diff --git a/app/commands/logbook.py b/app/commands/logbook.py index 01d5e81..98ce304 100644 --- a/app/commands/logbook.py +++ b/app/commands/logbook.py @@ -3,18 +3,13 @@ import click from datetime import datetime -from app.collect.logbook import update_entries as logbook_update_entries -from app.collect.takeoff_landings import update_entries as takeoff_landings_update_entries -from app.model import Airport, Logbook -from sqlalchemy.sql import func +from app.collect.logbook import update_takeoff_landings, update_logbook from tqdm import tqdm from app.commands.database import get_database_days from app.utils import date_to_timestamps -from app import db - user_cli = AppGroup("logbook") -user_cli.help = "Handling of logbook data." +user_cli.help = "Handling of takeoff/landings and logbook data." @user_cli.command("compute_takeoff_landing") @@ -29,7 +24,7 @@ def compute_takeoff_landing(start, end): for single_date in pbar: pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d")) (start, end) = date_to_timestamps(single_date) - result = takeoff_landings_update_entries(session=db.session, start=start, end=end) + result = update_takeoff_landings(start=start, end=end) @user_cli.command("compute_logbook") @@ -43,77 +38,4 @@ def compute_logbook(start, end): pbar = tqdm(days) for single_date in pbar: pbar.set_description(single_date.strftime("%Y-%m-%d")) - result = logbook_update_entries(session=db.session, date=single_date) - - -@user_cli.command("show") -@click.argument("airport_name") -@click.argument("date") -def show(airport_name, date=None): - """Show a logbook for .""" - airport = db.session.query(Airport).filter(Airport.name == airport_name).first() - - if airport is None: - print('Airport "{}" not found.'.format(airport_name)) - return - - or_args = [] - if date is not None: - date = datetime.strptime(date, "%Y-%m-%d") - (start, end) = date_to_timestamps(date) - or_args = [db.between(Logbook.reftime, start, end)] - - # get all logbook entries and add device and airport infos - logbook_query = ( - db.session.query(func.row_number().over(order_by=Logbook.reftime).label("row_number"), Logbook) - .filter(*or_args) - .filter(db.or_(Logbook.takeoff_airport_id == airport.id, Logbook.landing_airport_id == airport.id)) - .order_by(Logbook.reftime) - ) - - # ... and finally print out the logbook - print("--- Logbook ({}) ---".format(airport_name)) - - def none_datetime_replacer(datetime_object): - return "--:--:--" if datetime_object is None else datetime_object.time() - - def none_track_replacer(track_object): - return "--" if track_object is None else round(track_object / 10.0) - - def none_timedelta_replacer(timedelta_object): - return "--:--:--" if timedelta_object is None else timedelta_object - - def none_registration_replacer(device_object): - return "[" + device_object.address + "]" if len(device_object.infos) == 0 else device_object.infos[0].registration - - def none_aircraft_replacer(device_object): - return "(unknown)" if len(device_object.infos) == 0 else device_object.infos[0].aircraft - - def airport_marker(logbook_object): - if logbook_object.takeoff_airport is not None and logbook_object.takeoff_airport.name is not airport.name: - return "FROM: {}".format(logbook_object.takeoff_airport.name) - elif logbook_object.landing_airport is not None and logbook_object.landing_airport.name is not airport.name: - return "TO: {}".format(logbook_object.landing_airport.name) - else: - return "" - - def none_altitude_replacer(logbook_object): - return "?" if logbook_object.max_altitude is None else "{:5d}m ({:+5d}m)".format(logbook_object.max_altitude, logbook_object.max_altitude - logbook_object.takeoff_airport.altitude) - - for [row_number, logbook] in logbook_query.all(): - print( - "%3d. %10s %8s (%2s) %8s (%2s) %8s %15s %8s %17s %20s" - % ( - row_number, - logbook.reftime.date(), - none_datetime_replacer(logbook.takeoff_timestamp), - none_track_replacer(logbook.takeoff_track), - none_datetime_replacer(logbook.landing_timestamp), - none_track_replacer(logbook.landing_track), - none_timedelta_replacer(logbook.duration), - none_altitude_replacer(logbook), - none_registration_replacer(logbook.device), - none_aircraft_replacer(logbook.device), - airport_marker(logbook), - ) - ) + result = update_logbook(date=single_date) diff --git a/app/commands/stats.py b/app/commands/stats.py deleted file mode 100644 index 227aa8e..0000000 --- a/app/commands/stats.py +++ /dev/null @@ -1,130 +0,0 @@ -from flask.cli import AppGroup -import click - -from datetime import datetime -from tqdm import tqdm - -from app.commands.database import get_database_days - -from app.collect.stats import ( - create_device_stats, - create_receiver_stats, - create_relation_stats, - create_country_stats, - update_qualities, - update_receivers as update_receivers_command, - update_devices as update_devices_command, - update_device_stats_jumps, -) - -from app.collect.ognrange import update_entries as update_receiver_coverages -from app.model import Device - -from app import db - - -user_cli = AppGroup("stats") -user_cli.help = "Handling of statistical data." - - -@user_cli.command("create") -@click.argument("start") -@click.argument("end") -def create(start, end): - """Create DeviceStats, ReceiverStats and RelationStats.""" - - days = get_database_days(start, end) - - pbar = tqdm(days) - for single_date in pbar: - pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d")) - result = create_device_stats(session=db.session, date=single_date) - result = update_device_stats_jumps(session=db.session, date=single_date) - result = create_receiver_stats(session=db.session, date=single_date) - result = create_relation_stats(session=db.session, date=single_date) - result = update_qualities(session=db.session, date=single_date) - - -@user_cli.command("create_country") -@click.argument("start") -@click.argument("end") -def create_country(start, end): - """Create CountryStats.""" - - days = get_database_days(start, end) - - pbar = tqdm(days) - for single_date in pbar: - pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d")) - result = create_country_stats(session=db.session, date=single_date) - - -@user_cli.command("update_devices_name") -def update_devices_name(): - """Update Devices name.""" - - device_ids = db.session.query(Device.id).all() - - for device_id in tqdm(device_ids): - db.session.execute( - "update devices d set name = sq.name from ( select * from aircraft_beacons ab where ab.device_id = {} limit 1) sq where d.id = sq.device_id and d.name is null or d.name = 'ICA3D3CC4';".format( - device_id[0] - ) - ) - db.session.commit() - - -@user_cli.command("update_receivers") -def update_receivers(): - """Update receivers with data from stats.""" - - result = update_receivers_command(session=db.session) - print(result) - - -@user_cli.command("update_devices") -def update_devices(): - """Update devices with data from stats.""" - - result = update_devices_command(session=db.session) - print(result) - - -@user_cli.command("update_mgrs") -@click.argument("start") -@click.argument("end") -def update_mgrs(start, end): - """Create location_mgrs_short.""" - - days = get_database_days(start, end) - - pbar = tqdm(days) - for single_date in pbar: - datestr = datetime.strftime(single_date, "%Y-%m-%d") - pbar.set_description(datestr) - for pbar2 in tqdm(["{:02d}:{:02d}".format(hh, mm) for hh in range(0, 24) for mm in range(0, 60)]): - sql = """ - UPDATE aircraft_beacons - SET location_mgrs_short = left(location_mgrs, 5) || substring(location_mgrs, 6, 2) || substring(location_mgrs, 11, 2) - WHERE timestamp BETWEEN '{0} {1}:00' and '{0} {1}:59' AND location_mgrs_short IS NULL; - """.format( - datestr, pbar2 - ) - - # print(sql) - db.session.execute(sql) - db.session.commit() - - -@user_cli.command("create_ognrange") -@click.argument("start") -@click.argument("end") -def create_ognrange(start=None, end=None): - """Create receiver coverage stats for Melissas ognrange.""" - - days = get_database_days(start, end) - - pbar = tqdm(days) - for single_date in pbar: - pbar.set_description(datetime.strftime(single_date, "%Y-%m-%d")) - result = update_receiver_coverages(session=db.session, date=single_date) diff --git a/app/config/__init__.py b/app/config/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/app/config/default.py b/app/config/default.py deleted file mode 100644 index 439ca42..0000000 --- a/app/config/default.py +++ /dev/null @@ -1,28 +0,0 @@ -import os - -SECRET_KEY = "i-like-ogn" - -SQLALCHEMY_DATABASE_URI = os.environ.get("SQLALCHEMY_DATABASE_URI", "postgresql://postgres@localhost:5432/ogn") -SQLALCHEMY_TRACK_MODIFICATIONS = False - -# Flask-Cache stuff -CACHE_TYPE = "simple" -CACHE_DEFAULT_TIMEOUT = 300 - -# Celery stuff -CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379/0") -CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND", "redis://localhost:6379/0") - -from celery.schedules import crontab -from datetime import timedelta - -CELERYBEAT_SCHEDULE = { - "update-ddb": {"task": "import_ddb", "schedule": timedelta(hours=1)}, - "update-country-codes": {"task": "update_receivers_country_code", "schedule": timedelta(days=1)}, - "update-takeoff-and-landing": {"task": "update_takeoff_landings", "schedule": timedelta(hours=1), "kwargs": {"last_minutes": 90}}, - "update-logbook": {"task": "update_logbook_entries", "schedule": timedelta(hours=2), "kwargs": {"day_offset": 0}}, - "update-max-altitudes": {"task": "update_logbook_max_altitude", "schedule": timedelta(hours=1), "kwargs": {"day_offset": 0}}, - "update-stats-daily": {"task": "update_stats", "schedule": crontab(hour=0, minute=5), "kwargs": {"day_offset": -1}}, - "update-logbook-daily": {"task": "update_logbook_entries", "schedule": crontab(hour=1, minute=0), "kwargs": {"day_offset": -1}}, - "purge_old_data": {"task": "purge_old_data", "schedule": timedelta(hours=1), "kwargs": {"max_hours": 48}}, -} diff --git a/app/config/test.py b/app/config/test.py deleted file mode 100644 index 7024064..0000000 --- a/app/config/test.py +++ /dev/null @@ -1,7 +0,0 @@ -SQLALCHEMY_DATABASE_URI = "postgresql://postgres@localhost:5432/ogn_test" -SQLALCHEMY_TRACK_MODIFICATIONS = False -SQLALCHEMY_ECHO = True - -# Celery stuff -CELERY_BROKER_URL = "redis://localhost:6379/0" -CELERY_RESULT_BACKEND = "redis://localhost:6379/0" diff --git a/app/gateway/beacon_conversion.py b/app/gateway/beacon_conversion.py new file mode 100644 index 0000000..7925bf2 --- /dev/null +++ b/app/gateway/beacon_conversion.py @@ -0,0 +1,52 @@ +from flask import current_app + +from mgrs import MGRS + +from ogn.parser import parse + +from app.model import AircraftType + +#import rasterio as rs +#elevation_dataset = rs.open('/Volumes/LaCieBlack/Wtf4.tiff') + +mgrs = MGRS() + + +def aprs_string_to_message(aprs_string): + try: + message = parse(aprs_string, calculate_relations=True) + except Exception as e: + current_app.logger.debug(e) + return None + + if message['aprs_type'] not in ('position', 'status'): + return None + + elif message['aprs_type'] == 'position': + latitude = message["latitude"] + longitude = message["longitude"] + + message["location"] = "SRID=4326;POINT({} {})".format(longitude, latitude) + + location_mgrs = mgrs.toMGRS(latitude, longitude).decode("utf-8") + message["location_mgrs"] = location_mgrs + message["location_mgrs_short"] = location_mgrs[0:5] + location_mgrs[5:7] + location_mgrs[10:12] + + #if 'altitude' in message and longitude >= 0.0 and longitude <= 20.0 and latitude >= 40.0 and latitude <= 60.0: + # elevation = [val[0] for val in elevation_dataset.sample(((longitude, latitude),))][0] + # message['agl'] = message['altitude'] - elevation + + if 'bearing' in message: + bearing = int(message['bearing']) + message['bearing'] = bearing if bearing < 360 else 0 + + if "aircraft_type" in message: + message["aircraft_type"] = AircraftType(message["aircraft_type"]) if message["aircraft_type"] in AircraftType.list() else AircraftType.UNKNOWN + + if "gps_quality" in message: + if message["gps_quality"] is not None and "horizontal" in message["gps_quality"]: + message["gps_quality_horizontal"] = message["gps_quality"]["horizontal"] + message["gps_quality_vertical"] = message["gps_quality"]["vertical"] + del message["gps_quality"] + + return message diff --git a/app/gateway/bulkimport.py b/app/gateway/bulkimport.py deleted file mode 100644 index 4e0a88f..0000000 --- a/app/gateway/bulkimport.py +++ /dev/null @@ -1,206 +0,0 @@ -import os -import re -from datetime import datetime, timedelta -from io import StringIO - -from flask import current_app -from flask.cli import AppGroup -import click -from tqdm import tqdm -from mgrs import MGRS - -from ogn.parser import parse, ParseError - -from app.model import AircraftType, Location -from app.gateway.process_tools import open_file, create_tables, drop_tables, update_aircraft_beacons_bigdata - -from app import db - -user_cli = AppGroup("bulkimport") -user_cli.help = "Tools for accelerated data import." - - -basepath = os.path.dirname(os.path.realpath(__file__)) - -# define message types we want to proceed -AIRCRAFT_BEACON_TYPES = ["aprs_aircraft", "flarm", "tracker", "fanet", "lt24", "naviter", "skylines", "spider", "spot", "flymaster"] -RECEIVER_BEACON_TYPES = ["aprs_receiver", "receiver"] - -# define fields we want to proceed -BEACON_KEY_FIELDS = ["name", "receiver_name", "timestamp"] -AIRCRAFT_BEACON_FIELDS = [ - "location", - "altitude", - "dstcall", - "relay", - "track", - "ground_speed", - "address_type", - "aircraft_type", - "stealth", - "address", - "climb_rate", - "turn_rate", - "signal_quality", - "error_count", - "frequency_offset", - "gps_quality_horizontal", - "gps_quality_vertical", - "software_version", - "hardware_version", - "real_address", - "signal_power", - "distance", - "radial", - "quality", - "location_mgrs", - "location_mgrs_short", - "agl", -] -RECEIVER_BEACON_FIELDS = [ - "location", - "altitude", - "dstcall", -] - - -def initial_file_scan(file): - """Scan file and get rowcount and first server timestamp.""" - - row_count = 0 - timestamp = None - - for row in file: - row_count += 1 - if timestamp is None and row[0] == '#': - message = parse(row) - if message['aprs_type'] == 'server': - timestamp = message['timestamp'] - - file.seek(0) - return row_count, timestamp - - -class DbFeeder: - def __init__(self, postfix, reference_timestamp, auto_update_timestamp): - self.postfix = postfix - self.reference_timestamp = reference_timestamp - self.auto_update_timestamp = auto_update_timestamp - - self.last_flush = datetime.utcnow() - - self.aircraft_buffer = StringIO() - self.receiver_buffer = StringIO() - - self.connection = db.engine.raw_connection() - self.cursor = self.connection.cursor() - - self.mgrs = MGRS() - - create_tables(self.postfix) - - def __enter__(self): - return self - - def __exit__(self, *args): - self._flush() - update_aircraft_beacons_bigdata(self.postfix) - self.connection.commit() - - self.cursor.close() - self.connection.close() - - def _flush(self): - self.aircraft_buffer.seek(0) - self.receiver_buffer.seek(0) - - self.cursor.copy_from(self.aircraft_buffer, "aircraft_beacons_{postfix}".format(postfix=self.postfix), sep=",", columns=BEACON_KEY_FIELDS + AIRCRAFT_BEACON_FIELDS) - self.cursor.copy_from(self.receiver_buffer, "receiver_beacons_{postfix}".format(postfix=self.postfix), sep=",", columns=BEACON_KEY_FIELDS + RECEIVER_BEACON_FIELDS) - self.connection.commit() - - self.aircraft_buffer = StringIO() - self.receiver_buffer = StringIO() - - def add(self, raw_string): - try: - message = parse(raw_string, reference_timestamp=self.reference_timestamp) - except NotImplementedError as e: - current_app.logger.error("No parser implemented for message: {}".format(raw_string)) - return - except ParseError as e: - current_app.logger.error("Parsing error with message: {}".format(raw_string)) - return - except TypeError as e: - current_app.logger.error("TypeError with message: {}".format(raw_string)) - return - except Exception as e: - current_app.logger.error("Other Exception with string: {}".format(raw_string)) - return - - if message['aprs_type'] not in ('server', 'position'): - return - - elif message['aprs_type'] == 'server' and self.auto_update_timestamp is True: - self.reference_timestamp = message['timestamp'] - return - - elif message['aprs_type'] == 'position': - latitude = message["latitude"] - longitude = message["longitude"] - - location = Location(longitude, latitude) - message["location"] = location.to_wkt() - - location_mgrs = self.mgrs.toMGRS(latitude, longitude).decode("utf-8") - message["location_mgrs"] = location_mgrs - message["location_mgrs_short"] = location_mgrs[0:5] + location_mgrs[5:7] + location_mgrs[10:12] - - if "aircraft_type" in message: - message["aircraft_type"] = AircraftType(message["aircraft_type"]).name if message["aircraft_type"] in AircraftType.list() else AircraftType.UNKNOWN.name - - if "gps_quality" in message: - if message["gps_quality"] is not None and "horizontal" in message["gps_quality"]: - message["gps_quality_horizontal"] = message["gps_quality"]["horizontal"] - message["gps_quality_vertical"] = message["gps_quality"]["vertical"] - del message["gps_quality"] - - if message["beacon_type"] in RECEIVER_BEACON_TYPES: - complete_message = ",".join([str(message[k]) if k in message and message[k] is not None else "\\N" for k in BEACON_KEY_FIELDS + RECEIVER_BEACON_FIELDS]) - self.receiver_buffer.write(complete_message) - self.receiver_buffer.write("\n") - elif message["beacon_type"] in AIRCRAFT_BEACON_TYPES: - complete_message = ",".join([str(message[k]) if k in message and message[k] is not None else "\\N" for k in BEACON_KEY_FIELDS + AIRCRAFT_BEACON_FIELDS]) - self.aircraft_buffer.write(complete_message) - self.aircraft_buffer.write("\n") - else: - current_app.logger.error("Ignore beacon_type: {}".format(message["beacon_type"])) - return - - if datetime.utcnow() - self.last_flush >= timedelta(seconds=5): - self._flush() - self.last_flush = datetime.utcnow() - - -def convert(sourcefile): - with open_file(sourcefile) as filehandler: - total_lines, reference_timestamp = initial_file_scan(filehandler) - - if reference_timestamp is not None: - auto_update_timestamp = True - postfix = str(reference_timestamp.total_seconds()) - else: - auto_update_timestamp = False - match = re.match(r".*OGN_log\.txt_([0-9]{4}\-[0-9]{2}\-[0-9]{2})\.gz$", sourcefile) - if match: - reference_timestamp = datetime.strptime(match.group(1), "%Y-%m-%d") - postfix = reference_timestamp.strftime("%Y_%m_%d") - else: - current_app.logger.error("No reference time information. Skipping file: {}".format(sourcefile)) - return - - with open_file(sourcefile) as fin: - with DbFeeder(postfix=postfix, reference_timestamp=reference_timestamp, auto_update_timestamp=auto_update_timestamp) as feeder: - pbar = tqdm(fin, total=total_lines) - for line in pbar: - pbar.set_description("Importing {}".format(sourcefile)) - feeder.add(raw_string=line) diff --git a/app/gateway/message_handling.py b/app/gateway/message_handling.py new file mode 100644 index 0000000..bef9986 --- /dev/null +++ b/app/gateway/message_handling.py @@ -0,0 +1,438 @@ +import os +import time +from io import StringIO + +from app import db +from app.model import AircraftType +from app.utils import get_sql_trustworthy + +basepath = os.path.dirname(os.path.realpath(__file__)) + +# define fields we want to proceed +SENDER_POSITION_BEACON_FIELDS = [ + "reference_timestamp", + + "name", + "dstcall", + "relay", + "receiver_name", + "timestamp", + "location", + + "track", + "ground_speed", + "altitude", + + "address_type", + "aircraft_type", + "stealth", + "address", + "climb_rate", + "turn_rate", + "signal_quality", + "error_count", + "frequency_offset", + "gps_quality_horizontal", + "gps_quality_vertical", + "software_version", + "hardware_version", + "real_address", + "signal_power", + + "distance", + "bearing", + "normalized_quality", + + "location_mgrs", + "location_mgrs_short", + "agl", +] + +RECEIVER_POSITION_BEACON_FIELDS = [ + "reference_timestamp", + + "name", + "dstcall", + "receiver_name", + "timestamp", + "location", + + "altitude", + + "location_mgrs", + "location_mgrs_short", + "agl", +] + +RECEIVER_STATUS_BEACON_FIELDS = [ + "reference_timestamp", + + "name", + "dstcall", + "receiver_name", + "timestamp", + + "version", + "platform", + + "cpu_temp", + "rec_input_noise", +] + + +def sender_position_message_to_csv_string(message, none_character=''): + """ + Convert sender_position_messages to csv string. + + :param dict message: dict of sender position messages from the parser + :param str none_character: '' for a file, '\\N' for Postgresql COPY + """ + + csv_string = "{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11},{12},{13},{14},{15},{16},{17},{18},{19},{20},{21},{22},{23},{24},{25},{26},{27},{28},{29},{30}\n".format( + message['reference_timestamp'], + + message['name'], + message['dstcall'], + message['relay'] if 'relay' in message and message['relay'] else none_character, + message['receiver_name'], + message['timestamp'], + message['location'], + + message['track'] if 'track' in message and message['track'] else none_character, + message['ground_speed'] if 'ground_speed' in message and message['ground_speed'] else none_character, + int(message['altitude']) if message['altitude'] else none_character, + + message['address_type'] if 'address_type' in message and message['address_type'] else none_character, # 10 + message['aircraft_type'].name if 'aircraft_type' in message and message['aircraft_type'] else AircraftType.UNKNOWN.name, + message['stealth'] if 'stealth' in message and message['stealth'] else none_character, + message['address'] if 'address' in message and message['address'] else none_character, + message['climb_rate'] if 'climb_rate' in message and message['climb_rate'] else none_character, + message['turn_rate'] if 'turn_rate' in message and message['turn_rate'] else none_character, + message['signal_quality'] if 'signal_quality' in message and message['signal_quality'] else none_character, + message['error_count'] if 'error_count' in message and message['error_count'] else none_character, + message['frequency_offset'] if 'frequency_offset' in message and message['frequency_offset'] else none_character, + message['gps_quality_horizontal'] if 'gps_quality_horizontal' in message and message['gps_quality_horizontal'] else none_character, + message['gps_quality_vertical'] if 'gps_quality_vertical' in message and message['gps_quality_vertical'] else none_character, # 20 + message['software_version'] if 'software_version' in message and message['software_version'] else none_character, + message['hardware_version'] if 'hardware_version' in message and message['hardware_version'] else none_character, + message['real_address'] if 'real_address' in message and message['real_address'] else none_character, + message['signal_power'] if 'signal_power' in message and message['signal_power'] else none_character, + + message['distance'] if 'distance' in message and message['distance'] else none_character, + message['bearing'] if 'bearing' in message and message['bearing'] else none_character, + message['normalized_quality'] if 'normalized_quality' in message and message['normalized_quality'] else none_character, + + message['location_mgrs'], + message['location_mgrs_short'], + message['agl'] if 'agl' in message else none_character, + ) + return csv_string + + +def receiver_position_message_to_csv_string(message, none_character=''): + csv_string = "{0},{1},{2},{3},{4},{5},{6},{7},{8},{9}\n".format( + message['reference_timestamp'], + + message['name'], + message['dstcall'], + message['receiver_name'], + message['timestamp'], + message['location'], + + int(message['altitude']) if message['altitude'] else none_character, + + message['location_mgrs'], + message['location_mgrs_short'], + message['agl'] if 'agl' in message else none_character, + ) + return csv_string + + +def receiver_status_message_to_csv_string(message, none_character=''): + csv_string = "{0},{1},{2},{3},{4},{5},{6},{7},{8}\n".format( + message['reference_timestamp'], + + message['name'], + message['dstcall'], + message['receiver_name'], + message['timestamp'], + + message['version'] if 'version' in message else none_character, + message['platform'] if 'platform' in message else none_character, + + message['cpu_temp'] if 'cpu_temp' in message else none_character, + message['rec_input_noise'] if 'rec_input_noise' in message else none_character, + + ) + return csv_string + + +def sender_position_csv_strings_to_db(lines): + timestamp_string = str(time.time()).replace('.', '_') + tmp_tablename = f'sender_positions_{timestamp_string}' + + connection = db.engine.raw_connection() + cursor = connection.cursor() + + string_buffer = StringIO() + string_buffer.writelines(lines) + string_buffer.seek(0) + + cursor.execute(f"CREATE TEMPORARY TABLE {tmp_tablename} (LIKE sender_positions) ON COMMIT DROP;") + cursor.copy_from(file=string_buffer, table=tmp_tablename, sep=",", columns=SENDER_POSITION_BEACON_FIELDS) + + # Update agl + cursor.execute(f""" + UPDATE {tmp_tablename} AS tmp + SET + agl = tmp.altitude - ST_Value(e.rast, tmp.location) + FROM elevation AS e + WHERE ST_Intersects(tmp.location, e.rast); + """) + + # Update sender position statistics + cursor.execute(f""" + INSERT INTO sender_position_statistics AS sps (date, dstcall, address_type, aircraft_type, stealth, software_version, hardware_version, messages_count) + SELECT + tmp.reference_timestamp::DATE AS date, + tmp.dstcall, + tmp.address_type, + tmp.aircraft_type, + tmp.stealth, + tmp.software_version, + tmp.hardware_version, + COUNT(tmp.*) AS messages_count + FROM {tmp_tablename} AS tmp + GROUP BY date, dstcall, address_type, aircraft_type, stealth, software_version, hardware_version + ON CONFLICT (date, dstcall, address_type, aircraft_type, stealth, software_version, hardware_version) DO UPDATE + SET + messages_count = EXCLUDED.messages_count + sps.messages_count; + """) + + # Update senders + cursor.execute(f""" + INSERT INTO senders AS s (firstseen, lastseen, name, aircraft_type, stealth, address, software_version, hardware_version, real_address) + SELECT DISTINCT ON (tmp.name) + tmp.reference_timestamp AS firstseen, + tmp.reference_timestamp AS lastseen, + + tmp.name, + + tmp.aircraft_type, + tmp.stealth, + tmp.address, + tmp.software_version, + tmp.hardware_version, + tmp.real_address + FROM {tmp_tablename} AS tmp + WHERE tmp.name NOT LIKE 'RND%' + ON CONFLICT (name) DO UPDATE + SET + lastseen = GREATEST(EXCLUDED.lastseen, s.lastseen), + aircraft_type = EXCLUDED.aircraft_type, + stealth = EXCLUDED.stealth, + address = EXCLUDED.address, + software_version = COALESCE(EXCLUDED.software_version, s.software_version), + hardware_version = COALESCE(EXCLUDED.hardware_version, s.hardware_version), + real_address = COALESCE(EXCLUDED.real_address, s.real_address); + """) + + # Update sender_infos FK -> senders + cursor.execute(""" + UPDATE sender_infos AS si + SET sender_id = s.id + FROM senders AS s + WHERE si.sender_id IS NULL AND s.address = si.address; + """) + + SQL_TRUSTWORTHY = get_sql_trustworthy(source_table_alias='tmp') + + # Update coverage statistics + cursor.execute(f""" + INSERT INTO coverage_statistics AS rs (date, location_mgrs_short, sender_id, receiver_id, is_trustworthy, max_distance, max_normalized_quality, messages_count) + SELECT + tmp.reference_timestamp::DATE AS date, + tmp.location_mgrs_short, + tmp.sender_id, + tmp.receiver_id, + + ({SQL_TRUSTWORTHY}) AS is_trustworthy, + + MAX(tmp.distance) AS max_distance, + MAX(tmp.normalized_quality) AS max_normalized_quality, + COUNT(tmp.*) AS messages_count + FROM (SELECT x.*, s.id AS sender_id, r.id AS receiver_id FROM {tmp_tablename} AS x INNER JOIN senders AS s ON x.name = s.name INNER JOIN receivers AS r ON x.receiver_name = r.name) AS tmp + GROUP BY date, location_mgrs_short, sender_id, receiver_id, is_trustworthy + ON CONFLICT (date, location_mgrs_short, sender_id, receiver_id, is_trustworthy) DO UPDATE + SET + max_distance = GREATEST(EXCLUDED.max_distance, rs.max_distance), + max_normalized_quality = GREATEST(EXCLUDED.max_normalized_quality, rs.max_normalized_quality), + messages_count = EXCLUDED.messages_count + rs.messages_count; + """) + + # Insert all the beacons + all_fields = ', '.join(SENDER_POSITION_BEACON_FIELDS) + cursor.execute(f""" + INSERT INTO sender_positions ({all_fields}) + SELECT {all_fields} FROM {tmp_tablename}; + """) + + connection.commit() + + cursor.close() + connection.close() + + +def receiver_position_csv_strings_to_db(lines): + timestamp_string = str(time.time()).replace('.', '_') + tmp_tablename = f'receiver_positions_{timestamp_string}' + + connection = db.engine.raw_connection() + cursor = connection.cursor() + + string_buffer = StringIO() + string_buffer.writelines(lines) + string_buffer.seek(0) + + cursor.execute(f"CREATE TEMPORARY TABLE {tmp_tablename} (LIKE receiver_positions) ON COMMIT DROP;") + cursor.copy_from(file=string_buffer, table=tmp_tablename, sep=",", columns=RECEIVER_POSITION_BEACON_FIELDS) + + # Update agl + cursor.execute(f""" + UPDATE {tmp_tablename} AS tmp + SET + agl = tmp.altitude - ST_Value(e.rast, tmp.location) + FROM elevation AS e + WHERE ST_Intersects(tmp.location, e.rast); + """) + + # Update receivers + cursor.execute(f""" + INSERT INTO receivers AS r (firstseen, lastseen, name, timestamp, location, altitude, agl) + SELECT DISTINCT ON (tmp.name) + tmp.reference_timestamp AS firstseen, + tmp.reference_timestamp AS lastseen, + + tmp.name, + tmp.timestamp, + tmp.location, + + tmp.altitude, + + tmp.agl + FROM {tmp_tablename} AS tmp, + ( + SELECT + tmp.name, + MAX(timestamp) AS timestamp + FROM {tmp_tablename} AS tmp + GROUP BY tmp.name + ) AS sq + WHERE tmp.name = sq.name AND tmp.timestamp = sq.timestamp AND tmp.name NOT LIKE 'RND%' + ON CONFLICT (name) DO UPDATE + SET + lastseen = EXCLUDED.lastseen, + timestamp = EXCLUDED.timestamp, + location = EXCLUDED.location, + altitude = EXCLUDED.altitude, + + agl = EXCLUDED.agl; + """) + + # Update receiver country + cursor.execute(""" + UPDATE receivers AS r + SET + country_id = c.gid + FROM countries AS c + WHERE r.country_id IS NULL AND ST_Within(r.location, c.geom); + """) + + # Update receiver airport + cursor.execute(""" + UPDATE receivers AS r + SET + airport_id = ( + SELECT id + FROM airports AS a + WHERE + ST_Contains(a.border, r.location) + AND a.style IN (2,4,5) + ORDER BY ST_DistanceSphere(a.location, r.location) + LIMIT 1 + ) + WHERE r.airport_id IS NULL; + """) + + # Insert all the beacons + all_fields = ', '.join(RECEIVER_POSITION_BEACON_FIELDS) + cursor.execute(f""" + INSERT INTO receiver_positions ({all_fields}) + SELECT {all_fields} FROM {tmp_tablename}; + """) + + connection.commit() + + cursor.close() + connection.close() + + +def receiver_status_csv_strings_to_db(lines): + timestamp_string = str(time.time()).replace('.', '_') + tmp_tablename = f'receiver_status_{timestamp_string}' + + connection = db.engine.raw_connection() + cursor = connection.cursor() + + string_buffer = StringIO() + string_buffer.writelines(lines) + string_buffer.seek(0) + + cursor.execute(f"CREATE TEMPORARY TABLE {tmp_tablename} (LIKE receiver_statuses) ON COMMIT DROP;") + cursor.copy_from(file=string_buffer, table=tmp_tablename, sep=",", columns=RECEIVER_STATUS_BEACON_FIELDS) + + # Update receivers + cursor.execute(f""" + INSERT INTO receivers AS r (firstseen, lastseen, name, timestamp, version, platform, cpu_temp, rec_input_noise) + SELECT DISTINCT ON (tmp.name) + tmp.reference_timestamp AS firstseen, + tmp.reference_timestamp AS lastseen, + + tmp.name, + tmp.timestamp, + + tmp.version, + tmp.platform, + + tmp.cpu_temp, + tmp.rec_input_noise + FROM {tmp_tablename} AS tmp, + ( + SELECT + tmp.name, + MAX(timestamp) AS timestamp + FROM {tmp_tablename} AS tmp + GROUP BY tmp.name + ) AS sq + WHERE tmp.name = sq.name AND tmp.timestamp = sq.timestamp + ON CONFLICT (name) DO UPDATE + SET + lastseen = EXCLUDED.lastseen, + timestamp = EXCLUDED.timestamp, + version = EXCLUDED.version, + platform = EXCLUDED.platform, + cpu_temp = EXCLUDED.cpu_temp, + rec_input_noise = EXCLUDED.rec_input_noise; + """) + + # Insert all the beacons + all_fields = ', '.join(RECEIVER_STATUS_BEACON_FIELDS) + cursor.execute(f""" + INSERT INTO receiver_statuses ({all_fields}) + SELECT {all_fields} FROM {tmp_tablename}; + """) + + connection.commit() + + cursor.close() + connection.close() diff --git a/app/gateway/process_tools.py b/app/gateway/process_tools.py index 9cc46f0..1ec2bc1 100644 --- a/app/gateway/process_tools.py +++ b/app/gateway/process_tools.py @@ -38,73 +38,10 @@ class Timer(object): print("Elapsed: {}".format(time.time() - self.tstart)) -def drop_tables(postfix): - """Drop tables for log file import.""" - - db.session.execute(""" - DROP TABLE IF EXISTS "aircraft_beacons_{postfix}"; - DROP TABLE IF EXISTS "receiver_beacons_{postfix}"; - """.format(postfix=postfix)) - db.session.commit() - - -def create_tables(postfix): - """Create tables for log file import.""" - - drop_tables(postfix) - db.session.execute(""" - CREATE TABLE aircraft_beacons_{postfix} AS TABLE aircraft_beacons WITH NO DATA; - CREATE TABLE receiver_beacons_{postfix} AS TABLE receiver_beacons WITH NO DATA; - """.format(postfix=postfix)) - db.session.commit() - - -def update_aircraft_beacons_bigdata(postfix): - """Calculates distance/radial and quality and computes the altitude above ground level. - Due to performance reasons we use a new table instead of updating the old.""" - - db.session.execute(""" - SELECT - ab.location, ab.altitude, ab.name, ab.dstcall, ab.relay, ab.receiver_name, ab.timestamp, ab.track, ab.ground_speed, - - ab.address_type, ab.aircraft_type, ab.stealth, ab.address, ab.climb_rate, ab.turn_rate, ab.signal_quality, ab.error_count, - ab.frequency_offset, ab.gps_quality_horizontal, ab.gps_quality_vertical, ab.software_version, ab.hardware_version, ab.real_address, ab.signal_power, - - ab.location_mgrs, - ab.location_mgrs_short, - - CASE WHEN ab.location IS NOT NULL AND r.location IS NOT NULL THEN CAST(ST_DistanceSphere(ab.location, r.location) AS REAL) ELSE NULL END AS distance, - CASE WHEN ab.location IS NOT NULL AND r.location IS NOT NULL THEN CAST(degrees(ST_Azimuth(ab.location, r.location)) AS SMALLINT) % 360 ELSE NULL END AS radial, - CASE WHEN ab.location IS NOT NULL AND r.location IS NOT NULL AND ST_DistanceSphere(ab.location, r.location) > 0 AND ab.signal_quality IS NOT NULL - THEN CAST(signal_quality + 20*log(ST_DistanceSphere(ab.location, r.location)/10000) AS REAL) - ELSE NULL - END AS quality, - CAST((ab.altitude - subtable.elev_m) AS REAL) AS agl - INTO aircraft_beacons_{postfix}_temp - FROM - aircraft_beacons_{postfix} AS ab - JOIN LATERAL ( - SELECT ab.location, MAX(ST_NearestValue(e.rast, ab.location)) as elev_m - FROM elevation e - WHERE ST_Intersects(ab.location, e.rast) - GROUP BY ab.location - ) AS subtable ON TRUE, - (SELECT name, last(location, timestamp) AS location FROM receiver_beacons_{postfix} GROUP BY name) AS r - WHERE ab.receiver_name = r.name; - - DROP TABLE IF EXISTS "aircraft_beacons_{postfix}"; - ALTER TABLE "aircraft_beacons_{postfix}_temp" RENAME TO "aircraft_beacons_{postfix}"; - """.format(postfix=postfix)) - - -def export_to_path(postfix, path): +def export_to_path(path): connection = db.engine.raw_connection() cursor = connection.cursor() - aircraft_beacons_file = os.path.join(path, "aircraft_beacons_{postfix}.csv.gz".format(postfix=postfix)) + aircraft_beacons_file = os.path.join(path, "sender_positions.csv.gz") with gzip.open(aircraft_beacons_file, "wt", encoding="utf-8") as gzip_file: - cursor.copy_expert("COPY ({}) TO STDOUT WITH (DELIMITER ',', FORMAT CSV, HEADER, ENCODING 'UTF-8');".format("SELECT * FROM aircraft_beacons_{postfix}".format(postfix=postfix)), gzip_file) - - receiver_beacons_file = os.path.join(path, "receiver_beacons_{postfix}.csv.gz".format(postfix=postfix)) - with gzip.open(receiver_beacons_file, "wt") as gzip_file: - cursor.copy_expert("COPY ({}) TO STDOUT WITH (DELIMITER ',', FORMAT CSV, HEADER, ENCODING 'UTF-8');".format("SELECT * FROM receiver_beacons_{postfix}".format(postfix=postfix)), gzip_file) + cursor.copy_expert("COPY ({}) TO STDOUT WITH (DELIMITER ',', FORMAT CSV, HEADER, ENCODING 'UTF-8');".format("SELECT * FROM sender_positions"), gzip_file) diff --git a/app/main/__init__.py b/app/main/__init__.py index 7e35f16..5829369 100644 --- a/app/main/__init__.py +++ b/app/main/__init__.py @@ -3,3 +3,4 @@ from flask import Blueprint bp = Blueprint("main", __name__) import app.main.routes +import app.main.jinja_filters diff --git a/app/main/jinja_filters.py b/app/main/jinja_filters.py new file mode 100644 index 0000000..6cdb960 --- /dev/null +++ b/app/main/jinja_filters.py @@ -0,0 +1,64 @@ +from app.main import bp +from app.model import Airport, Sender, Receiver + +from flask import url_for +import time +import datetime +import math + + +@bp.app_template_filter() +def timestamp_to_status(timestamp): + if datetime.datetime.utcnow() - timestamp < datetime.timedelta(minutes=10): + return 'OK' + elif datetime.datetime.utcnow() - timestamp < datetime.timedelta(hours=1): + return '?' + else: + return 'OFFLINE' + + +@bp.app_template_filter() +def to_html_link(obj): + if isinstance(obj, Airport): + airport = obj + return f"""{airport.name}""" + + elif isinstance(obj, Sender): + sender = obj + if len(sender.infos) > 0 and len(sender.infos[0].registration) > 0: + return f"""{sender.infos[0].registration}""" + elif sender.address: + return f"""[{sender.address}]""" + else: + return f"""[{sender.name}]""" + + elif isinstance(obj, Receiver): + receiver = obj + return f"""{receiver.name}""" + + elif obj is None: + return "-" + + else: + raise NotImplementedError("cant apply filter 'to_html_link' to object {type(obj)}") + + +@bp.app_template_filter() +def to_ordinal(rad): + deg = math.degrees(rad) + if deg >= 337.5 or deg < 22.5: + return "N" + elif deg >= 22.5 and deg < 67.5: + return "NW" + elif deg >= 67.5 and deg < 112.5: + return "W" + elif deg >= 112.5 and deg < 157.5: + return "SW" + elif deg >= 157.5 and deg < 202.5: + return "S" + elif deg >= 202.5 and deg < 247.5: + return "SE" + elif deg >= 247.5 and deg < 292.5: + return "E" + elif deg >= 292.5 and deg < 337.5: + return "NE" diff --git a/app/main/live_routes.py b/app/main/live_routes.py deleted file mode 100644 index 9c6a621..0000000 --- a/app/main/live_routes.py +++ /dev/null @@ -1,87 +0,0 @@ -from flask import request, render_template, current_app -from flask_cors import cross_origin - -from app.backend.liveglidernet import rec, lxml -from app.main import bp - - -@bp.route("/live.html") -@cross_origin() -def live(): - return render_template("ogn_live.html", host=request.host) - - -@bp.route("/rec.php") -def rec_php(): - a = request.args.get("a") - z = request.args.get("z") - - xml = rec() - resp = current_app.make_response(xml) - resp.mimetype = "text/xml" - return resp - - -@bp.route("/lxml.php") -def lxml_php(): - a = request.args.get("a") - b = request.args.get("b") - c = request.args.get("c") - d = request.args.get("d") - e = request.args.get("e") - z = request.args.get("z") - - xml = lxml() - resp = current_app.make_response(xml) - resp.mimetype = "text/xml" - return resp - - -@bp.route("/pict/") -def pict(filename): - return current_app.send_static_file("ognlive/pict/" + filename) - - -@bp.route("/favicon.gif") -def favicon_gif(): - return current_app.send_static_file("ognlive/pict/favicon.gif") - - -@bp.route("/horizZoomControl.js") -def horizZoomControl_js(): - return current_app.send_static_file("ognlive/horizZoomControl.js") - - -@bp.route("/barogram.js") -def barogram_js(): - return current_app.send_static_file("ognlive/barogram.js") - - -@bp.route("/util.js") -def util_js(): - return current_app.send_static_file("ognlive/util.js") - - -@bp.route("/ogn.js") -def ogn_js(): - return current_app.send_static_file("ognlive/ogn.js") - - -@bp.route("/ol.js") -def ol_js(): - return current_app.send_static_file("ognlive/ol.js") - - -@bp.route("/osm.js") -def osm_js(): - return current_app.send_static_file("ognlive/osm.js") - - -@bp.route("/ol.css") -def ol_css(): - return current_app.send_static_file("ognlive/ol.css") - - -@bp.route("/osm.css") -def osm_css(): - return current_app.send_static_file("ognlive/osm.css") diff --git a/app/main/matplotlib_service.py b/app/main/matplotlib_service.py new file mode 100644 index 0000000..7b4cf54 --- /dev/null +++ b/app/main/matplotlib_service.py @@ -0,0 +1,44 @@ +from app import db +from app.model import SenderDirectionStatistic +import random +import numpy as np +import matplotlib.pyplot as plt +from matplotlib.figure import Figure + + +def create_range_figure2(sender_id): + fig = Figure() + axis = fig.add_subplot(1, 1, 1) + xs = range(100) + ys = [random.randint(1, 50) for x in xs] + axis.plot(xs, ys) + + return fig + + +def create_range_figure(sender_id): + sds = db.session.query(SenderDirectionStatistic) \ + .filter(SenderDirectionStatistic.sender_id == sender_id) \ + .order_by(SenderDirectionStatistic.directions_count.desc()) \ + .limit(1) \ + .one() + + fig = Figure() + + direction_data = sds.direction_data + max_range = max([r['max_range'] / 1000.0 for r in direction_data]) + + theta = np.array([i['direction'] / 180 * np.pi for i in direction_data]) + radii = np.array([i['max_range'] / 1000 if i['max_range'] > 0 else 0 for i in direction_data]) + width = np.array([13 / 180 * np.pi for i in direction_data]) + colors = plt.cm.viridis(radii / max_range) + + ax = fig.add_subplot(111, projection='polar') + ax.bar(theta, radii, width=width, bottom=0.0, color=colors, edgecolor='b', alpha=0.5) + #ax.set_rticks([0, 25, 50, 75, 100, 125, 150]) + ax.set_theta_zero_location("N") + ax.set_theta_direction(-1) + + fig.suptitle(f"Range between sender '{sds.sender.name}' and receiver '{sds.receiver.name}'") + + return fig diff --git a/app/main/routes.py b/app/main/routes.py index 550a737..67ea3d0 100644 --- a/app/main/routes.py +++ b/app/main/routes.py @@ -1,12 +1,13 @@ -import datetime +from datetime import date, time, datetime from flask import request, render_template, send_file from app import db from app import cache -from app.model import Airport, Country, Device, Logbook, Receiver, ReceiverStats +from app.model import Airport, Country, Sender, SenderInfo, TakeoffLanding, Logbook, Receiver, SenderPosition, RelationStatistic, ReceiverStatistic, SenderStatistic from app.main import bp +from app.main.matplotlib_service import create_range_figure @cache.cached(key_prefix="countries_in_receivers") @@ -16,28 +17,26 @@ def get_countries_in_receivers(): return [{"iso2": country[0]} for country in query.all()] -@cache.cached(key_prefix="countries_in_logbook") -def get_countries_in_logbook(): - query = db.session.query(Country.iso2).filter(Country.iso2 == Airport.country_code).filter(Logbook.takeoff_airport_id == Airport.id).order_by(Country.iso2).distinct(Country.iso2) - +@cache.cached(key_prefix="countries_in_takeoff_landings") +def get_used_countries(): + query = db.session.query(Country.iso2).filter(Country.gid == TakeoffLanding.country_id).order_by(Country.iso2).distinct(Country.iso2) return [{"iso2": country[0]} for country in query.all()] @cache.memoize() -def get_airports_in_country(sel_country): - query = db.session.query(Airport.id, Airport.name).filter(Airport.country_code == sel_country).filter(Logbook.takeoff_airport_id == Airport.id).order_by(Airport.name).distinct(Airport.name) - - return [{"id": airport[0], "name": airport[1]} for airport in query.all()] +def get_used_airports_by_country(sel_country): + query = db.session.query(Airport).filter(Airport.country_code == sel_country).filter(TakeoffLanding.airport_id == Airport.id).filter(TakeoffLanding.country_id == Country.gid).order_by(Airport.name).distinct(Airport.name) + return [used_airport for used_airport in query] @cache.memoize() def get_dates_for_airport(sel_airport): query = ( - db.session.query(db.func.date(Logbook.reftime), db.func.count(Logbook.id).label("logbook_count")) + db.session.query(db.func.date(Logbook.reference_timestamp), db.func.count(Logbook.id).label("logbook_count")) .filter(Airport.id == sel_airport) .filter(db.or_(Airport.id == Logbook.takeoff_airport_id, Airport.id == Logbook.landing_airport_id)) - .group_by(db.func.date(Logbook.reftime)) - .order_by(db.func.date(Logbook.reftime).desc()) + .group_by(db.func.date(Logbook.reference_timestamp)) + .order_by(db.func.date(Logbook.reference_timestamp).desc()) ) return [{"date": date, "logbook_count": logbook_count} for (date, logbook_count) in query.all()] @@ -46,21 +45,56 @@ def get_dates_for_airport(sel_airport): @bp.route("/") @bp.route("/index.html") def index(): - return render_template("base.html") + today_beginning = datetime.combine(date.today(), time()) + + senders_today = db.session.query(db.func.count(Sender.id)).filter(Sender.lastseen >= today_beginning).one()[0] + receivers_today = db.session.query(db.func.count(Receiver.id)).filter(Receiver.lastseen >= today_beginning).one()[0] + takeoffs_today = db.session.query(db.func.count(TakeoffLanding.id)).filter(db.and_(TakeoffLanding.timestamp >= today_beginning, TakeoffLanding.is_takeoff is True)).one()[0] + landings_today = db.session.query(db.func.count(TakeoffLanding.id)).filter(db.and_(TakeoffLanding.timestamp >= today_beginning, TakeoffLanding.is_takeoff is False)).one()[0] + sender_positions_today = db.session.query(db.func.sum(ReceiverStatistic.messages_count)).filter(ReceiverStatistic.date == date.today()).one()[0] + sender_positions_total = db.session.query(db.func.sum(ReceiverStatistic.messages_count)).one()[0] + + last_logbook_entries = db.session.query(Logbook).order_by(Logbook.reference_timestamp.desc()).limit(10) + return render_template( + "index.html", + senders_today=senders_today, + receivers_today=receivers_today, + takeoffs_today=takeoffs_today, + landings_today=landings_today, + sender_positions_today=sender_positions_today, + sender_positions_total=sender_positions_total, + logbook=last_logbook_entries) -@bp.route("/devices.html", methods=["GET", "POST"]) -def devices(): - devices = db.session.query(Device).order_by(Device.address) - return render_template("devices.html", devices=devices) +@bp.route("/senders.html", methods=["GET", "POST"]) +def senders(): + senders = db.session.query(Sender) \ + .options(db.joinedload(Sender.infos)) \ + .order_by(Sender.name) + return render_template("senders.html", senders=senders) -@bp.route("/device_detail.html", methods=["GET", "POST"]) -def device_detail(): - device_id = request.args.get("id") - device = db.session.query(Device).filter(Device.id == device_id).one() +@bp.route("/sender_detail.html", methods=["GET", "POST"]) +def sender_detail(): + sender_id = request.args.get("sender_id") + sender = db.session.query(Sender).filter(Sender.id == sender_id).one() - return render_template("device_detail.html", title="Device", device=device) + return render_template("sender_detail.html", title="Sender", sender=sender) + + +@bp.route("/range_view.png") +def range_view(): + import io + from flask import Response + + from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas + + sender_id = request.args.get("sender_id") + + fig = create_range_figure(sender_id) + output = io.BytesIO() + FigureCanvas(fig).print_png(output) + return Response(output.getvalue(), mimetype='image/png') @bp.route("/receivers.html") @@ -71,41 +105,34 @@ def receivers(): # Get receiver selection list if sel_country: - receivers = db.session.query(Receiver).filter(db.and_(Receiver.country_id == Country.gid, Country.iso2 == sel_country)).order_by(Receiver.name) + receivers = db.session.query(Receiver) \ + .options(db.joinedload(Receiver.airport)) \ + .filter(db.and_(Receiver.country_id == Country.gid, Country.iso2 == sel_country)) \ + .order_by(Receiver.name) else: - receivers = db.session.query(Receiver).order_by(Receiver.name) + receivers = db.session.query(Receiver) \ + .options(db.joinedload(Receiver.airport)) \ + .order_by(Receiver.name) return render_template("receivers.html", title="Receivers", sel_country=sel_country, countries=countries, receivers=receivers) @bp.route("/receiver_detail.html") def receiver_detail(): - sel_receiver_id = request.args.get("receiver_id") + receiver_id = request.args.get("receiver_id") - receiver = db.session.query(Receiver).filter(Receiver.id == sel_receiver_id).one() - - airport = ( - db.session.query(Airport) - .filter( - db.and_( - Receiver.id == sel_receiver_id, - db.func.st_contains(db.func.st_buffer(Receiver.location_wkt, 0.5), Airport.location_wkt), - db.func.st_distance_sphere(Airport.location_wkt, Receiver.location_wkt) < 1000, - ) - ) - .filter(Airport.style.in_((2, 4, 5))) - ) - return render_template("receiver_detail.html", title="Receiver Detail", receiver=receiver, airport=airport.first()) + receiver = db.session.query(Receiver).filter(Receiver.id == receiver_id).one() + return render_template("receiver_detail.html", title="Receiver Detail", receiver=receiver) @bp.route("/airports.html", methods=["GET", "POST"]) def airports(): sel_country = request.args.get("country") - countries = get_countries_in_logbook() + countries = get_used_countries() if sel_country: - airports = get_airports_in_country(sel_country) + airports = get_used_airports_by_country(sel_country) else: airports = [] @@ -116,41 +143,41 @@ def airports(): @bp.route("/airport_detail.html") def airport_detail(): - sel_airport = request.args.get("airport") + sel_airport = request.args.get("airport_id") airport = db.session.query(Airport).filter(Airport.id == sel_airport) - devices = db.session.query(Device).join(Logbook).filter(Logbook.takeoff_airport_id == sel_airport).order_by(Device.address) + senders = db.session.query(Sender).join(Logbook).filter(Logbook.takeoff_airport_id == sel_airport).order_by(Sender.name) - return render_template("airport_detail.html", title="Airport Detail", airport=airport.one(), devices=devices) + return render_template("airport_detail.html", title="Airport Detail", airport=airport.one(), senders=senders) -@bp.route("/logbook.html", methods=["GET", "POST"]) -def logbook(): +@bp.route("/logbooks.html", methods=["GET", "POST"]) +def logbooks(): sel_country = request.args.get("country") - sel_airport = request.args.get("airport") + sel_airport_id = request.args.get("airport_id") sel_date = request.args.get("date") - sel_device_id = request.args.get("device_id") + sel_sender_id = request.args.get("sender_id") - countries = get_countries_in_logbook() + countries = get_used_countries() if sel_country: - airports = get_airports_in_country(sel_country) + airports = get_used_airports_by_country(sel_country) else: airports = [] - if sel_airport: - sel_airport = int(sel_airport) - if sel_airport not in [airport["id"] for airport in airports]: - sel_airport = None + if sel_airport_id: + sel_airport_id = int(sel_airport_id) + if sel_airport_id not in [airport.id for airport in airports]: + sel_airport_id = None sel_date = None - dates = get_dates_for_airport(sel_airport) + dates = get_dates_for_airport(sel_airport_id) else: dates = [] if sel_date: - sel_date = datetime.datetime.strptime(sel_date, "%Y-%m-%d").date() + sel_date = datetime.strptime(sel_date, "%Y-%m-%d").date() if sel_date not in [entry["date"] for entry in dates]: sel_date = dates[0]["date"] elif len(dates) > 0: @@ -158,21 +185,21 @@ def logbook(): # Get Logbook filters = [] - if sel_airport: - filters.append(db.or_(Logbook.takeoff_airport_id == sel_airport, Logbook.landing_airport_id == sel_airport)) + if sel_airport_id: + filters.append(db.or_(Logbook.takeoff_airport_id == sel_airport_id, Logbook.landing_airport_id == sel_airport_id)) if sel_date: - filters.append(db.func.date(Logbook.reftime) == sel_date) + filters.append(db.func.date(Logbook.reference_timestamp) == sel_date) - if sel_device_id: - filters.append(Logbook.device_id == sel_device_id) + if sel_sender_id: + filters.append(Logbook.sender_id == sel_sender_id) if len(filters) > 0: - logbook = db.session.query(Logbook).filter(*filters).order_by(Logbook.reftime) + logbooks = db.session.query(Logbook).filter(*filters).order_by(Logbook.reference_timestamp).limit(100) else: - logbook = None + logbooks = None - return render_template("logbook.html", title="Logbook", sel_country=sel_country, countries=countries, sel_airport=sel_airport, airports=airports, sel_date=sel_date, dates=dates, logbook=logbook) + return render_template("logbooks.html", title="Logbook", sel_country=sel_country, countries=countries, sel_airport_id=sel_airport_id, airports=airports, sel_date=sel_date, dates=dates, logbooks=logbooks) @bp.route("/download.html") @@ -186,12 +213,27 @@ def download_flight(): return send_file(buffer, as_attachment=True, attachment_filename="wtf.igc", mimetype="text/plain") -@bp.route("/statistics.html") -def statistics(): +@bp.route("/sender_ranking.html") +def sender_ranking(): + sender_statistics = db.session.query(SenderStatistic) \ + .filter(db.and_(SenderStatistic.date == date.today(), SenderStatistic.is_trustworthy is True)) \ + .order_by(SenderStatistic.max_distance.desc()) \ + .all() - today = datetime.date.today() - today = datetime.date(2018, 7, 31) + return render_template( + "sender_ranking.html", + title="Sender Ranking", + ranking=sender_statistics) - receiverstats = db.session.query(ReceiverStats).filter(ReceiverStats.date == today) - return render_template("statistics.html", title="Receiver Statistics", receiverstats=receiverstats) +@bp.route("/receiver_ranking.html") +def receiver_ranking(): + receiver_statistics = db.session.query(ReceiverStatistic) \ + .filter(db.and_(ReceiverStatistic.date == date.today(), ReceiverStatistic.is_trustworthy is True)) \ + .order_by(ReceiverStatistic.max_distance.desc()) \ + .all() + + return render_template( + "receiver_ranking.html", + title="Receiver Ranking", + ranking=receiver_statistics) diff --git a/app/model/__init__.py b/app/model/__init__.py index dcd1818..89fe239 100644 --- a/app/model/__init__.py +++ b/app/model/__init__.py @@ -1,21 +1,22 @@ # flake8: noqa from .aircraft_type import AircraftType -from .beacon import Beacon from .country import Country -from .country_stats import CountryStats -from .device import Device -from .device_info import DeviceInfo -from .device_info_origin import DeviceInfoOrigin -from .device_stats import DeviceStats -from .aircraft_beacon import AircraftBeacon -from .receiver_beacon import ReceiverBeacon +from .sender import Sender +from .sender_info_origin import SenderInfoOrigin +from .sender_info import SenderInfo +from .sender_position import SenderPosition +from .receiver_position import ReceiverPosition +from .receiver_status import ReceiverStatus from .receiver import Receiver -from .receiver_stats import ReceiverStats from .takeoff_landing import TakeoffLanding from .airport import Airport from .logbook import Logbook -from .receiver_coverage import ReceiverCoverage -from .relation_stats import RelationStats -from .flights2d import Flight2D from .geo import Location + +from .relation_statistic import RelationStatistic +from .coverage_statistic import CoverageStatistic +from .sender_statistic import SenderStatistic +from .receiver_statistic import ReceiverStatistic +from .sender_position_statistic import SenderPositionStatistic +from .sender_direction_statistic import SenderDirectionStatistic diff --git a/app/model/aircraft_beacon.py b/app/model/aircraft_beacon.py deleted file mode 100644 index abfbe67..0000000 --- a/app/model/aircraft_beacon.py +++ /dev/null @@ -1,130 +0,0 @@ -from sqlalchemy.sql import func -from app import db - -from .beacon import Beacon -from .aircraft_type import AircraftType - - -class AircraftBeacon(Beacon): - __tablename__ = "aircraft_beacons" - - # Flarm specific data - address_type = db.Column(db.SmallInteger) - aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN) - stealth = db.Column(db.Boolean) - address = db.Column(db.String) - climb_rate = db.Column(db.Float(precision=2)) - turn_rate = db.Column(db.Float(precision=2)) - signal_quality = db.Column(db.Float(precision=2)) - error_count = db.Column(db.SmallInteger) - frequency_offset = db.Column(db.Float(precision=2)) - gps_quality_horizontal = db.Column(db.SmallInteger) - gps_quality_vertical = db.Column(db.SmallInteger) - software_version = db.Column(db.Float(precision=2)) - hardware_version = db.Column(db.SmallInteger) - real_address = db.Column(db.String(6)) - signal_power = db.Column(db.Float(precision=2)) - proximity = None - - # Calculated values - distance = db.Column(db.Float(precision=2)) - radial = db.Column(db.SmallInteger) - quality = db.Column(db.Float(precision=2)) # signal quality normalized to 10km - location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars) - location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool - agl = db.Column(db.Float(precision=2)) - - def __repr__(self): - return "" % ( - self.address_type, - self.aircraft_type, - self.stealth, - self.address, - self.climb_rate, - self.turn_rate, - self.signal_quality, - self.error_count, - self.frequency_offset, - self.gps_quality_horizontal, - self.gps_quality_vertical, - self.software_version, - self.hardware_version, - self.real_address, - self.signal_power, - self.distance, - self.radial, - self.quality, - self.location_mgrs, - self.location_mgrs_short, - ) - - @classmethod - def get_columns(self): - return [ - "location", - "altitude", - "name", - "dstcall", - "relay", - "receiver_name", - "timestamp", - "track", - "ground_speed", - # 'raw_message', - # 'reference_timestamp', - "address_type", - "aircraft_type", - "stealth", - "address", - "climb_rate", - "turn_rate", - "signal_quality", - "error_count", - "frequency_offset", - "gps_quality_horizontal", - "gps_quality_vertical", - "software_version", - "hardware_version", - "real_address", - "signal_power", - "distance", - "radial", - "quality", - "location_mgrs", - "location_mgrs_short", - ] - - def get_values(self): - return [ - self.location_wkt, - int(self.altitude) if self.altitude else None, - self.name, - self.dstcall, - self.relay, - self.receiver_name, - self.timestamp, - self.track, - self.ground_speed, - # self.raw_message, - # self.reference_timestamp, - self.address_type, - self.aircraft_type, - self.stealth, - self.address, - self.climb_rate, - self.turn_rate, - self.signal_quality, - self.error_count, - self.frequency_offset, - self.gps_quality_horizontal, - self.gps_quality_vertical, - self.software_version, - self.hardware_version, - self.real_address, - self.signal_power, - self.distance, - self.radial, - self.quality, - self.location_mgrs, - self.location_mgrs_short, - ] diff --git a/app/model/beacon.py b/app/model/beacon.py deleted file mode 100644 index c23ebeb..0000000 --- a/app/model/beacon.py +++ /dev/null @@ -1,45 +0,0 @@ -from geoalchemy2.shape import to_shape -from geoalchemy2.types import Geometry -from sqlalchemy.ext.declarative import AbstractConcreteBase -from sqlalchemy.ext.hybrid import hybrid_property - -from .geo import Location - -from app import db - - -class Beacon(AbstractConcreteBase, db.Model): - # APRS data - location_wkt = db.Column("location", Geometry("POINT", srid=4326)) - altitude = db.Column(db.Float(precision=2)) - - name = db.Column(db.String, primary_key=True, nullable=True) - dstcall = db.Column(db.String) - relay = db.Column(db.String) - receiver_name = db.Column(db.String(9), primary_key=True, nullable=True) - timestamp = db.Column(db.DateTime, primary_key=True, nullable=True) - symboltable = None - symbolcode = None - track = db.Column(db.SmallInteger) - ground_speed = db.Column(db.Float(precision=2)) - comment = None - - # Type information - beacon_type = None - aprs_type = None - - # Debug information - raw_message = None - reference_timestamp = None - - @hybrid_property - def location(self): - if self.location_wkt is None: - return None - - coords = to_shape(self.location_wkt) - return Location(lat=coords.y, lon=coords.x) - - @location.expression - def location(cls): - return cls.location_wkt diff --git a/app/model/country_stats.py b/app/model/country_stats.py deleted file mode 100644 index 9f5c9bc..0000000 --- a/app/model/country_stats.py +++ /dev/null @@ -1,17 +0,0 @@ -from app import db - - -class CountryStats(db.Model): - __tablename__ = "country_stats" - - id = db.Column(db.Integer, primary_key=True) - - date = db.Column(db.Date) - - # Static data - aircraft_beacon_count = db.Column(db.Integer) - device_count = db.Column(db.Integer) - - # Relations - country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="SET NULL"), index=True) - country = db.relationship("Country", foreign_keys=[country_id], backref=db.backref("stats", order_by="CountryStats.date.asc()")) diff --git a/app/model/coverage_statistic.py b/app/model/coverage_statistic.py new file mode 100644 index 0000000..8fa39d9 --- /dev/null +++ b/app/model/coverage_statistic.py @@ -0,0 +1,25 @@ +from app import db + + +class CoverageStatistic(db.Model): + __tablename__ = "coverage_statistics" + + id = db.Column(db.Integer, primary_key=True) + + date = db.Column(db.Date) + location_mgrs_short = db.Column(db.String(9)) + is_trustworthy = db.Column(db.Boolean) + + messages_count = db.Column(db.Integer) + max_distance = db.Column(db.Float(precision=2)) + max_normalized_quality = db.Column(db.Float(precision=2)) + coverages_count = db.Column(db.Integer) + + # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("coverage_stats", order_by=date)) + + receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True) + receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("coverage_stats", order_by=date)) + + __table_args__ = (db.Index('idx_coverage_statistics_uc', 'date', 'location_mgrs_short', 'sender_id', 'receiver_id', 'is_trustworthy', unique=True), ) diff --git a/app/model/device_info_origin.py b/app/model/device_info_origin.py deleted file mode 100644 index a779697..0000000 --- a/app/model/device_info_origin.py +++ /dev/null @@ -1,8 +0,0 @@ -import enum - - -class DeviceInfoOrigin(enum.Enum): - UNKNOWN = 0 - OGN_DDB = 1 - FLARMNET = 2 - USER_DEFINED = 3 diff --git a/app/model/device_stats.py b/app/model/device_stats.py deleted file mode 100644 index 378be1e..0000000 --- a/app/model/device_stats.py +++ /dev/null @@ -1,52 +0,0 @@ -from app import db - -from .aircraft_type import AircraftType - - -class DeviceStats(db.Model): - __tablename__ = "device_stats" - - id = db.Column(db.Integer, primary_key=True) - - date = db.Column(db.Date) - - # Static data - name = db.Column(db.String) - firstseen = db.Column(db.DateTime) - lastseen = db.Column(db.DateTime) - aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN) - stealth = db.Column(db.Boolean) - software_version = db.Column(db.Float(precision=2)) - hardware_version = db.Column(db.SmallInteger) - real_address = db.Column(db.String(6)) - - # Statistic data - max_altitude = db.Column(db.Float(precision=2)) - receiver_count = db.Column(db.SmallInteger) - aircraft_beacon_count = db.Column(db.Integer) - jumps = db.Column(db.SmallInteger) - ambiguous = db.Column(db.Boolean) - quality = db.Column(db.Float(precision=2)) - - # Relation statistic data - quality_offset = db.Column(db.Float(precision=2)) - - # Ranking data - max_altitude_ranking_worldwide = db.Column(db.Integer) - max_altitude_ranking_country = db.Column(db.Integer) - receiver_count_ranking_worldwide = db.Column(db.Integer) - receiver_count_ranking_country = db.Column(db.Integer) - aircraft_beacon_count_ranking_worldwide = db.Column(db.Integer) - aircraft_beacon_count_ranking_country = db.Column(db.Integer) - quality_ranking_worldwide = db.Column(db.Integer) - quality_ranking_country = db.Column(db.Integer) - - # Relations - device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), index=True) - device = db.relationship("Device", foreign_keys=[device_id], backref=db.backref("stats", order_by="DeviceStats.date.asc()")) - - def __repr__(self): - return "" % (self.date, self.receiver_count, self.aircraft_beacon_count, self.max_altitude) - - -db.Index("ix_device_stats_date_device_id", DeviceStats.date, DeviceStats.device_id) diff --git a/app/model/flights2d.py b/app/model/flights2d.py deleted file mode 100644 index 8f47f06..0000000 --- a/app/model/flights2d.py +++ /dev/null @@ -1,24 +0,0 @@ -from geoalchemy2.types import Geometry - -from app import db - - -class Flight2D(db.Model): - __tablename__ = "flights2d" - - date = db.Column(db.Date, primary_key=True) - flight_type = db.Column(db.SmallInteger, primary_key=True) - - path_wkt = db.Column("path", Geometry("MULTILINESTRING", srid=4326)) - path_simple_wkt = db.Column("path_simple", Geometry("MULTILINESTRING", srid=4326)) # this is the path simplified with ST_Simplify(path, 0.0001) - - # Relations - device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), primary_key=True) - device = db.relationship("Device", foreign_keys=[device_id], backref="flights2d") - - def __repr__(self): - return "" % (self.date, self.path_wkt, self.path_simple_wkt) - - -db.Index("ix_flights2d_date_device_id", Flight2D.date, Flight2D.device_id) -# db.Index('ix_flights2d_date_path', Flight2D.date, Flight2D.path_wkt) --> CREATE INDEX ix_flights2d_date_path ON flights2d USING GIST("date", path) diff --git a/app/model/logbook.py b/app/model/logbook.py index 3c61772..c294a9e 100644 --- a/app/model/logbook.py +++ b/app/model/logbook.py @@ -1,14 +1,13 @@ from sqlalchemy.ext.hybrid import hybrid_property -from sqlalchemy.sql import null, case + from app import db class Logbook(db.Model): - __tablename__ = "logbook" + __tablename__ = "logbooks" id = db.Column(db.Integer, primary_key=True) - reftime = db.Column(db.DateTime, index=True) takeoff_timestamp = db.Column(db.DateTime) takeoff_track = db.Column(db.SmallInteger) landing_timestamp = db.Column(db.DateTime) @@ -16,14 +15,20 @@ class Logbook(db.Model): max_altitude = db.Column(db.Float(precision=2)) # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("logbook_entries", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc())) + takeoff_airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="CASCADE"), index=True) - takeoff_airport = db.relationship("Airport", foreign_keys=[takeoff_airport_id]) + takeoff_airport = db.relationship("Airport", foreign_keys=[takeoff_airport_id], backref=db.backref("logbook_entries_takeoff", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc())) + + takeoff_country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="CASCADE"), index=True) + takeoff_country = db.relationship("Country", foreign_keys=[takeoff_country_id], backref=db.backref("logbook_entries_takeoff", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc())) landing_airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="CASCADE"), index=True) - landing_airport = db.relationship("Airport", foreign_keys=[landing_airport_id]) + landing_airport = db.relationship("Airport", foreign_keys=[landing_airport_id], backref=db.backref("logbook_entries_landing", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc())) - device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="CASCADE"), index=True) - device = db.relationship("Device", foreign_keys=[device_id], backref=db.backref("logbook", order_by="Logbook.reftime")) + landing_country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="CASCADE"), index=True) + landing_country = db.relationship("Country", foreign_keys=[landing_country_id], backref=db.backref("logbook_entries_landing", order_by=db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null()).desc())) @hybrid_property def duration(self): @@ -31,4 +36,22 @@ class Logbook(db.Model): @duration.expression def duration(cls): - return case({False: None, True: cls.landing_timestamp - cls.takeoff_timestamp}, cls.landing_timestamp != null() and cls.takeoff_timestamp != null()) + return db.case({False: None, True: cls.landing_timestamp - cls.takeoff_timestamp}, cls.landing_timestamp != db.null() and cls.takeoff_timestamp != db.null()) + + @hybrid_property + def reference_timestamp(self): + return self.takeoff_timestamp if self.takeoff_timestamp is not None else self.landing_timestamp + + @reference_timestamp.expression + def reference_timestamp(cls): + return db.case({True: cls.takeoff_timestamp, False: cls.landing_timestamp}, cls.takeoff_timestamp != db.null()) + + #__table_args__ = (db.Index('idx_logbook_reference_timestamp', db.case({True: takeoff_timestamp, False: landing_timestamp}, takeoff_timestamp != db.null())),) + # FIXME: does not work... + +# FIXME: this does not throw an error as the __table_args__ above, but there is no index created +#_wrapped_case = f"({db.case(whens={True: Logbook.takeoff_timestamp, False: Logbook.landing_timestamp}, value=Logbook.takeoff_timestamp != db.null())})" +#Index("idx_logbook_reference_timestamp", _wrapped_case) + +# TODO: +# so execute manually: CREATE INDEX IF NOT EXISTS idx_logbook_reference_timestamp ON logbooks ((CASE takeoff_timestamp IS NULL WHEN true THEN takeoff_timestamp WHEN false THEN landing_timestamp END)); diff --git a/app/model/receiver.py b/app/model/receiver.py index cfae2a4..45a0a38 100644 --- a/app/model/receiver.py +++ b/app/model/receiver.py @@ -5,25 +5,37 @@ from .geo import Location from app import db +from .airport import Airport + class Receiver(db.Model): __tablename__ = "receivers" id = db.Column(db.Integer, primary_key=True) + name = db.Column(db.String(9)) location_wkt = db.Column("location", Geometry("POINT", srid=4326)) altitude = db.Column(db.Float(precision=2)) - name = db.Column(db.String(9), index=True) firstseen = db.Column(db.DateTime, index=True) lastseen = db.Column(db.DateTime, index=True) + timestamp = db.Column(db.DateTime, index=True) version = db.Column(db.String) platform = db.Column(db.String) + cpu_temp = db.Column(db.Float(precision=2)) + rec_input_noise = db.Column(db.Float(precision=2)) + + agl = db.Column(db.Float(precision=2)) # Relations country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="SET NULL"), index=True) country = db.relationship("Country", foreign_keys=[country_id], backref=db.backref("receivers", order_by="Receiver.name.asc()")) + airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="CASCADE"), index=True) + airport = db.relationship("Airport", foreign_keys=[airport_id], backref=db.backref("receivers", order_by="Receiver.name.asc()")) + + __table_args__ = (db.Index('idx_receivers_name_uc', 'name', unique=True), ) + @property def location(self): if self.location_wkt is None: @@ -31,3 +43,14 @@ class Receiver(db.Model): coords = to_shape(self.location_wkt) return Location(lat=coords.y, lon=coords.x) + + def airports_nearby(self): + query = ( + db.session.query(Airport, db.func.st_distance_sphere(self.location_wkt, Airport.location_wkt), db.func.st_azimuth(self.location_wkt, Airport.location_wkt)) + .filter(db.func.st_contains(db.func.st_buffer(Airport.location_wkt, 1), self.location_wkt)) + .filter(Airport.style.in_((2, 4, 5))) + .order_by(db.func.st_distance_sphere(self.location_wkt, Airport.location_wkt).asc()) + .limit(5) + ) + airports = [(airport, distance, azimuth) for airport, distance, azimuth in query] + return airports diff --git a/app/model/receiver_beacon.py b/app/model/receiver_beacon.py deleted file mode 100644 index 5aad4cb..0000000 --- a/app/model/receiver_beacon.py +++ /dev/null @@ -1,45 +0,0 @@ -from .beacon import Beacon - - -class ReceiverBeacon(Beacon): - __tablename__ = "receiver_beacons" - - # disable irrelevant aprs fields - relay = None - track = None - ground_speed = None - - def __repr__(self): - return "" % ( - self.name, - self.location, - self.altitude, - self.dstcall, - self.receiver_name, - self.timestamp, - ) - - @classmethod - def get_columns(self): - return [ - "location", - "altitude", - "name", - "dstcall", - "receiver_name", - "timestamp", - # 'raw_message', - # 'reference_timestamp', - ] - - def get_values(self): - return [ - self.location_wkt, - int(self.altitude) if self.altitude else None, - self.name, - self.dstcall, - self.receiver_name, - self.timestamp, - # self.raw_message, - # self.reference_timestamp, - ] diff --git a/app/model/receiver_coverage.py b/app/model/receiver_coverage.py deleted file mode 100644 index c0b543a..0000000 --- a/app/model/receiver_coverage.py +++ /dev/null @@ -1,23 +0,0 @@ -from app import db - - -class ReceiverCoverage(db.Model): - __tablename__ = "receiver_coverages" - - location_mgrs_short = db.Column(db.String(9), primary_key=True) - date = db.Column(db.Date, primary_key=True) - - max_signal_quality = db.Column(db.Float) - max_altitude = db.Column(db.Float(precision=2)) - min_altitude = db.Column(db.Float(precision=2)) - aircraft_beacon_count = db.Column(db.Integer) - - device_count = db.Column(db.SmallInteger) - - # Relations - receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="SET NULL"), primary_key=True) - receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("receiver_coverages", order_by="ReceiverCoverage.date.asc()")) - - -db.Index("ix_receiver_coverages_date_receiver_id", ReceiverCoverage.date, ReceiverCoverage.receiver_id) -db.Index("ix_receiver_coverages_receiver_id_date", ReceiverCoverage.receiver_id, ReceiverCoverage.date) diff --git a/app/model/receiver_position.py b/app/model/receiver_position.py new file mode 100644 index 0000000..401ceea --- /dev/null +++ b/app/model/receiver_position.py @@ -0,0 +1,39 @@ +from geoalchemy2.types import Geometry +from app import db + + +class ReceiverPosition(db.Model): + __tablename__ = "receiver_positions" + + reference_timestamp = db.Column(db.DateTime, primary_key=True) + + # APRS data + name = db.Column(db.String) + dstcall = db.Column(db.String) + #relay = db.Column(db.String) + receiver_name = db.Column(db.String(9)) + timestamp = db.Column(db.DateTime) + location = db.Column("location", Geometry("POINT", srid=4326)) + symboltable = None + symbolcode = None + + #track = db.Column(db.SmallInteger) + #ground_speed = db.Column(db.Float(precision=2)) + altitude = db.Column(db.Float(precision=2)) + + comment = None + + # Type information + beacon_type = None + aprs_type = None + + # Debug information + raw_message = None + + # Receiver specific data + user_comment = None + + # Calculated values (from this software) + location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars) + location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool + agl = db.Column(db.Float(precision=2)) diff --git a/app/model/receiver_statistic.py b/app/model/receiver_statistic.py new file mode 100644 index 0000000..1b20b59 --- /dev/null +++ b/app/model/receiver_statistic.py @@ -0,0 +1,22 @@ +from app import db + + +class ReceiverStatistic(db.Model): + __tablename__ = "receiver_statistics" + + id = db.Column(db.Integer, primary_key=True) + + date = db.Column(db.Date) + is_trustworthy = db.Column(db.Boolean) + + max_distance = db.Column(db.Float(precision=2)) + max_normalized_quality = db.Column(db.Float(precision=2)) + messages_count = db.Column(db.Integer) + coverages_count = db.Column(db.Integer) + senders_count = db.Column(db.Integer) + + # Relations + receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True) + receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("statistics", order_by=date.desc())) + + __table_args__ = (db.Index('idx_receiver_statistics_uc', 'date', 'receiver_id', 'is_trustworthy', unique=True), ) diff --git a/app/model/receiver_stats.py b/app/model/receiver_stats.py deleted file mode 100644 index 6001075..0000000 --- a/app/model/receiver_stats.py +++ /dev/null @@ -1,41 +0,0 @@ -from geoalchemy2.types import Geometry - -from app import db - - -class ReceiverStats(db.Model): - __tablename__ = "receiver_stats" - - id = db.Column(db.Integer, primary_key=True) - - date = db.Column(db.Date) - - # Static data - firstseen = db.Column(db.DateTime, index=True) - lastseen = db.Column(db.DateTime, index=True) - location_wkt = db.Column("location", Geometry("POINT", srid=4326)) - altitude = db.Column(db.Float(precision=2)) - version = db.Column(db.String) - platform = db.Column(db.String) - - # Statistic data - aircraft_beacon_count = db.Column(db.Integer) - aircraft_count = db.Column(db.SmallInteger) - max_distance = db.Column(db.Float) - quality = db.Column(db.Float(precision=2)) - - # Relation statistic data - quality_offset = db.Column(db.Float(precision=2)) - - # Ranking data - aircraft_beacon_count_ranking = db.Column(db.SmallInteger) - aircraft_count_ranking = db.Column(db.SmallInteger) - max_distance_ranking = db.Column(db.SmallInteger) - quality_ranking = db.Column(db.Integer) - - # Relations - receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="SET NULL"), index=True) - receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("stats", order_by="ReceiverStats.date.asc()")) - - -db.Index("ix_receiver_stats_date_receiver_id", ReceiverStats.date, ReceiverStats.receiver_id) diff --git a/app/model/receiver_status.py b/app/model/receiver_status.py new file mode 100644 index 0000000..d589dab --- /dev/null +++ b/app/model/receiver_status.py @@ -0,0 +1,48 @@ +from app import db + + +class ReceiverStatus(db.Model): + __tablename__ = "receiver_statuses" + + reference_timestamp = db.Column(db.DateTime, primary_key=True) + + # APRS data + name = db.Column(db.String) + dstcall = db.Column(db.String) + receiver_name = db.Column(db.String(9)) + timestamp = db.Column(db.DateTime) + + # Type information + beacon_type = None + aprs_type = None + + # Debug information + raw_message = None + + # Receiver specific data + version = db.Column(db.String) + platform = db.Column(db.String) + cpu_load = None + free_ram = None + total_ram = None + ntp_error = None + + rt_crystal_correction = None + voltage = None + amperage = None + cpu_temp = db.Column(db.Float(precision=2)) + senders_visible = None + senders_total = None + rec_crystal_correction = None + rec_crystal_correction_fine = None + rec_input_noise = db.Column(db.Float(precision=2)) + senders_signal = None + senders_messages = None + good_senders_signal = None + good_senders = None + good_and_bad_senders = None + + # Calculated values (from this software) + location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars) + location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool + agl = db.Column(db.Float(precision=2)) diff --git a/app/model/relation_statistic.py b/app/model/relation_statistic.py new file mode 100644 index 0000000..92dfeb6 --- /dev/null +++ b/app/model/relation_statistic.py @@ -0,0 +1,23 @@ +from app import db + + +class RelationStatistic(db.Model): + __tablename__ = "relation_statistics" + + id = db.Column(db.Integer, primary_key=True) + + date = db.Column(db.Date) + is_trustworthy = db.Column(db.Boolean) + + max_distance = db.Column(db.Float(precision=2)) + max_normalized_quality = db.Column(db.Float(precision=2)) + messages_count = db.Column(db.Integer) + + # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("relation_stats", order_by=date)) + + receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True) + receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("relation_stats", order_by=date)) + + __table_args__ = (db.Index('idx_relation_statistics_uc', 'date', 'sender_id', 'receiver_id', 'is_trustworthy', unique=True), ) diff --git a/app/model/relation_stats.py b/app/model/relation_stats.py deleted file mode 100644 index 2bddebb..0000000 --- a/app/model/relation_stats.py +++ /dev/null @@ -1,26 +0,0 @@ -from app import db - - -class RelationStats(db.Model): - __tablename__ = "relation_stats" - - id = db.Column(db.Integer, primary_key=True) - - date = db.Column(db.Date) - - # Statistic data - quality = db.Column(db.Float(precision=2)) - beacon_count = db.Column(db.Integer) - - # Relations - device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), index=True) - device = db.relationship("Device", foreign_keys=[device_id], backref="relation_stats") - receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="SET NULL"), index=True) - receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref="relation_stats") - - def __repr__(self): - return "" % (self.date, self.quality, self.beacon_count) - - -db.Index("ix_relation_stats_date_device_id", RelationStats.date, RelationStats.device_id, RelationStats.receiver_id) -db.Index("ix_relation_stats_date_receiver_id", RelationStats.date, RelationStats.receiver_id, RelationStats.device_id) diff --git a/app/model/device.py b/app/model/sender.py similarity index 70% rename from app/model/device.py rename to app/model/sender.py index e83f2fd..7273cb6 100644 --- a/app/model/device.py +++ b/app/model/sender.py @@ -3,18 +3,16 @@ import datetime from sqlalchemy.ext.hybrid import hybrid_property from app import db -from .device_info import DeviceInfo from app.model.aircraft_type import AircraftType -class Device(db.Model): - __tablename__ = "devices" +class Sender(db.Model): + __tablename__ = "senders" id = db.Column(db.Integer, primary_key=True) + name = db.Column(db.String) - name = db.Column(db.String, index=True) - # address = db.Column(db.String(6), index=True) - address = db.Column(db.String, index=True) + address = db.Column(db.String(6), index=True) firstseen = db.Column(db.DateTime, index=True) lastseen = db.Column(db.DateTime, index=True) aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN) @@ -23,21 +21,16 @@ class Device(db.Model): hardware_version = db.Column(db.SmallInteger) real_address = db.Column(db.String(6)) + __table_args__ = (db.Index('idx_senders_name_uc', 'name', unique=True), ) + def __repr__(self): - return "" % (self.address, self.aircraft_type, self.stealth, self.software_version, self.hardware_version, self.real_address) - - @hybrid_property - def info(self): - query = db.session.query(DeviceInfo).filter(DeviceInfo.address == self.address).order_by(DeviceInfo.address_origin) - - return query.first() - - def get_infos(self): - query = db.session.query(DeviceInfo).filter(DeviceInfo.address == self.address).order_by(DeviceInfo.address_origin) - - return [info for info in query.all()] + return "" % (self.address, self.aircraft_type, self.stealth, self.software_version, self.hardware_version, self.real_address) EXPIRY_DATES = { + 7.01: datetime.date(2022, 2, 28), + 7.0: datetime.date(2021, 10, 31), + 6.83: datetime.date(2021, 10, 31), + 6.82: datetime.date(2021, 5, 31), 6.81: datetime.date(2021, 1, 31), 6.80: datetime.date(2021, 1, 31), 6.67: datetime.date(2020, 10, 31), diff --git a/app/model/sender_direction_statistic.py b/app/model/sender_direction_statistic.py new file mode 100644 index 0000000..d468989 --- /dev/null +++ b/app/model/sender_direction_statistic.py @@ -0,0 +1,22 @@ +from app import db + +from sqlalchemy.dialects.postgresql import JSON + + +class SenderDirectionStatistic(db.Model): + __tablename__ = "sender_direction_statistics" + + id = db.Column(db.Integer, primary_key=True) + + directions_count = db.Column(db.Integer) + messages_count = db.Column(db.Integer) + direction_data = db.Column(db.JSON) + + # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("direction_stats", order_by=directions_count.desc())) + + receiver_id = db.Column(db.Integer, db.ForeignKey("receivers.id", ondelete="CASCADE"), index=True) + receiver = db.relationship("Receiver", foreign_keys=[receiver_id], backref=db.backref("direction_stats", order_by=directions_count.desc())) + + __table_args__ = (db.Index('idx_sender_direction_statistics_uc', 'sender_id', 'receiver_id', unique=True), ) diff --git a/app/model/device_info.py b/app/model/sender_info.py similarity index 55% rename from app/model/device_info.py rename to app/model/sender_info.py index 7b480c7..38e8f2b 100644 --- a/app/model/device_info.py +++ b/app/model/sender_info.py @@ -1,15 +1,16 @@ from app import db -from .device_info_origin import DeviceInfoOrigin +from .sender_info_origin import SenderInfoOrigin from .aircraft_type import AircraftType +#from sqlalchemy.dialects.postgresql import ENUM -class DeviceInfo(db.Model): - __tablename__ = "device_infos" + +class SenderInfo(db.Model): + __tablename__ = "sender_infos" id = db.Column(db.Integer, primary_key=True) + address = db.Column(db.String(6), index=True) address_type = None - # address = db.Column(db.String(6), index=True) - address = db.Column(db.String, index=True) aircraft = db.Column(db.String) registration = db.Column(db.String(7)) competition = db.Column(db.String(3)) @@ -17,10 +18,14 @@ class DeviceInfo(db.Model): identified = db.Column(db.Boolean) aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN) - address_origin = db.Column(db.Enum(DeviceInfoOrigin), nullable=False, default=DeviceInfoOrigin.UNKNOWN) + address_origin = db.Column(db.Enum(SenderInfoOrigin), nullable=False, default=SenderInfoOrigin.UNKNOWN) + + # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id"), index=True) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("infos", order_by=address_origin)) def __repr__(self): - return "" % ( + return "" % ( self.address_type, self.address, self.aircraft, diff --git a/app/model/sender_info_origin.py b/app/model/sender_info_origin.py new file mode 100644 index 0000000..eb7c4a0 --- /dev/null +++ b/app/model/sender_info_origin.py @@ -0,0 +1,9 @@ +import enum + + +class SenderInfoOrigin(enum.Enum): + # lower number == more trustworthy + USER_DEFINED = 0 + OGN_DDB = 1 + FLARMNET = 2 + UNKNOWN = 3 diff --git a/app/model/sender_position.py b/app/model/sender_position.py new file mode 100644 index 0000000..914736a --- /dev/null +++ b/app/model/sender_position.py @@ -0,0 +1,62 @@ +from geoalchemy2.types import Geometry +from app import db + +from .aircraft_type import AircraftType + + +class SenderPosition(db.Model): + __tablename__ = "sender_positions" + + reference_timestamp = db.Column(db.DateTime, primary_key=True) + + # APRS data + name = db.Column(db.String) + dstcall = db.Column(db.String) + relay = db.Column(db.String) + receiver_name = db.Column(db.String(9)) + timestamp = db.Column(db.DateTime) + location = db.Column("location", Geometry("POINT", srid=4326)) + symboltable = None + symbolcode = None + + track = db.Column(db.SmallInteger) + ground_speed = db.Column(db.Float(precision=2)) + altitude = db.Column(db.Float(precision=2)) + + comment = None + + # Type information + beacon_type = None + aprs_type = None + + # Debug information + raw_message = None + + # Flarm specific data + address_type = db.Column(db.SmallInteger) + aircraft_type = db.Column(db.Enum(AircraftType), nullable=False, default=AircraftType.UNKNOWN) + stealth = db.Column(db.Boolean) + address = db.Column(db.String) + climb_rate = db.Column(db.Float(precision=2)) + turn_rate = db.Column(db.Float(precision=2)) + signal_quality = db.Column(db.Float(precision=2)) + error_count = db.Column(db.SmallInteger) + frequency_offset = db.Column(db.Float(precision=2)) + gps_quality_horizontal = db.Column(db.SmallInteger) + gps_quality_vertical = db.Column(db.SmallInteger) + software_version = db.Column(db.Float(precision=2)) + hardware_version = db.Column(db.SmallInteger) + real_address = db.Column(db.String(6)) + signal_power = db.Column(db.Float(precision=2)) + + #proximity = None + + # Calculated values (from parser) + distance = db.Column(db.Float(precision=2)) + bearing = db.Column(db.SmallInteger) + normalized_quality = db.Column(db.Float(precision=2)) # signal quality normalized to 10km + + # Calculated values (from this software) + location_mgrs = db.Column(db.String(15)) # full mgrs (15 chars) + location_mgrs_short = db.Column(db.String(9)) # reduced mgrs (9 chars), e.g. used for melissas range tool + agl = db.Column(db.Float(precision=2)) diff --git a/app/model/sender_position_statistic.py b/app/model/sender_position_statistic.py new file mode 100644 index 0000000..8a92780 --- /dev/null +++ b/app/model/sender_position_statistic.py @@ -0,0 +1,24 @@ +from app import db + +from .aircraft_type import AircraftType + +from sqlalchemy.dialects.postgresql import ENUM + + +class SenderPositionStatistic(db.Model): + __tablename__ = "sender_position_statistics" + + id = db.Column(db.Integer, primary_key=True) + + date = db.Column(db.Date) + + dstcall = db.Column(db.String) + address_type = db.Column(db.SmallInteger) + aircraft_type = db.Column(ENUM(AircraftType, create_type=False), nullable=False, default=AircraftType.UNKNOWN) + stealth = db.Column(db.Boolean) + software_version = db.Column(db.Float(precision=2)) + hardware_version = db.Column(db.SmallInteger) + + messages_count = db.Column(db.Integer) + + __table_args__ = (db.Index('idx_sender_position_statistics_uc', 'date', 'dstcall', 'address_type', 'aircraft_type', 'stealth', 'software_version', 'hardware_version', unique=True), ) diff --git a/app/model/sender_statistic.py b/app/model/sender_statistic.py new file mode 100644 index 0000000..9721683 --- /dev/null +++ b/app/model/sender_statistic.py @@ -0,0 +1,22 @@ +from app import db + + +class SenderStatistic(db.Model): + __tablename__ = "sender_statistics" + + id = db.Column(db.Integer, primary_key=True) + + date = db.Column(db.Date) + is_trustworthy = db.Column(db.Boolean) + + max_distance = db.Column(db.Float(precision=2)) + max_normalized_quality = db.Column(db.Float(precision=2)) + messages_count = db.Column(db.Integer) + coverages_count = db.Column(db.Integer) + receivers_count = db.Column(db.Integer) + + # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE"), index=True) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref=db.backref("statistics", order_by=date.desc())) + + __table_args__ = (db.Index('idx_sender_statistics_uc', 'date', 'sender_id', 'is_trustworthy', unique=True), ) diff --git a/app/model/takeoff_landing.py b/app/model/takeoff_landing.py index 3bfa966..0598ca2 100644 --- a/app/model/takeoff_landing.py +++ b/app/model/takeoff_landing.py @@ -4,13 +4,20 @@ from app import db class TakeoffLanding(db.Model): __tablename__ = "takeoff_landings" - device_id = db.Column(db.Integer, db.ForeignKey("devices.id", ondelete="SET NULL"), primary_key=True) - airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="SET NULL"), primary_key=True) - timestamp = db.Column(db.DateTime, primary_key=True) + id = db.Column(db.Integer, primary_key=True) + timestamp = db.Column(db.DateTime) is_takeoff = db.Column(db.Boolean) track = db.Column(db.SmallInteger) # Relations + sender_id = db.Column(db.Integer, db.ForeignKey("senders.id", ondelete="CASCADE")) + sender = db.relationship("Sender", foreign_keys=[sender_id], backref="takeoff_landings") + + airport_id = db.Column(db.Integer, db.ForeignKey("airports.id", ondelete="SET NULL")) airport = db.relationship("Airport", foreign_keys=[airport_id], backref="takeoff_landings") - device = db.relationship("Device", foreign_keys=[device_id], backref="takeoff_landings", order_by="TakeoffLanding.timestamp") + + country_id = db.Column(db.Integer, db.ForeignKey("countries.gid", ondelete="CASCADE"), index=True) + country = db.relationship("Country", foreign_keys=[country_id], backref="takeoff_landings") + + __table_args__ = (db.Index('idx_takeoff_landings_uc', 'timestamp', 'sender_id', 'airport_id', unique=True), ) diff --git a/app/tasks/__init__.py b/app/tasks/__init__.py new file mode 100644 index 0000000..31f918a --- /dev/null +++ b/app/tasks/__init__.py @@ -0,0 +1,5 @@ +from .sql_tasks import update_statistics, update_sender_direction_statistics + +from .orm_tasks import transfer_to_database +from .orm_tasks import update_takeoff_landings, update_logbook, update_logbook_max_altitude +from .orm_tasks import import_ddb diff --git a/app/tasks/orm_tasks.py b/app/tasks/orm_tasks.py new file mode 100644 index 0000000..b1b5a57 --- /dev/null +++ b/app/tasks/orm_tasks.py @@ -0,0 +1,52 @@ +from datetime import datetime, timedelta + +from app.collect.logbook import update_takeoff_landings as logbook_update_takeoff_landings, update_logbook as logbook_update +from app.collect.logbook import update_max_altitudes as logbook_update_max_altitudes + +from app.collect.database import import_ddb as device_infos_import_ddb + +from app.collect.gateway import transfer_from_redis_to_database + +from app import db, celery + + +@celery.task(name="transfer_to_database") +def transfer_to_database(): + """Transfer APRS data from Redis to database.""" + + result = transfer_from_redis_to_database() + return result + + +@celery.task(name="update_takeoff_landings") +def update_takeoff_landings(last_minutes): + """Compute takeoffs and landings.""" + + end = datetime.utcnow() + start = end - timedelta(minutes=last_minutes) + result = logbook_update_takeoff_landings(start=start, end=end) + return result + + +@celery.task(name="update_logbook") +def update_logbook(offset_days=None): + """Add/update logbook entries.""" + + result = logbook_update(offset_days=offset_days) + return result + + +@celery.task(name="update_logbook_max_altitude") +def update_logbook_max_altitude(): + """Add max altitudes in logbook when flight is complete (takeoff and landing).""" + + result = logbook_update_max_altitudes() + return result + + +@celery.task(name="import_ddb") +def import_ddb(): + """Import registered devices from the DDB.""" + + result = device_infos_import_ddb() + return result diff --git a/app/tasks/sql_tasks.py b/app/tasks/sql_tasks.py new file mode 100644 index 0000000..294acba --- /dev/null +++ b/app/tasks/sql_tasks.py @@ -0,0 +1,126 @@ +from datetime import datetime, timedelta + +from app import db, celery + + +@celery.task(name="update_statistics") +def update_statistics(date_str=None): + """ Update relation_statistics, sender_statistics, receiver_statistics (all depend on coverage_statistics).""" + + if date_str is None: + date_str = datetime.utcnow().strftime("%Y-%m-%d") + + # Update relation statistics + db.session.execute(f""" + DELETE FROM relation_statistics + WHERE date = '{date_str}'; + + INSERT INTO relation_statistics AS rs (date, sender_id, receiver_id, is_trustworthy, max_distance, max_normalized_quality, messages_count, coverages_count) + SELECT + tmp.date, + tmp.sender_id, + tmp.receiver_id, + + is_trustworthy, + + MAX(tmp.max_distance) AS max_distance, + MAX(tmp.max_normalized_quality) AS max_normalized_quality, + SUM(tmp.messages_count) AS messages_count, + COUNT(DISTINCT tmp.location_mgrs_short) AS coverages_count + FROM coverage_statistics AS tmp + WHERE tmp.date = '{date_str}' + GROUP BY date, sender_id, receiver_id, is_trustworthy; + """) + + # Update sender statistics + db.session.execute(f""" + DELETE FROM sender_statistics + WHERE date = '{date_str}'; + + INSERT INTO sender_statistics AS rs (date, sender_id, is_trustworthy, max_distance, max_normalized_quality, messages_count, coverages_count, receivers_count) + SELECT + tmp.date, + tmp.sender_id, + + is_trustworthy, + + MAX(tmp.max_distance) AS max_distance, + MAX(tmp.max_normalized_quality) AS max_normalized_quality, + SUM(tmp.messages_count) AS messages_count, + COUNT(DISTINCT tmp.location_mgrs_short) AS coverages_count, + COUNT(DISTINCT tmp.receiver_id) AS receivers_count + FROM coverage_statistics AS tmp + WHERE tmp.date = '{date_str}' + GROUP BY date, sender_id, is_trustworthy; + """) + + # Update receiver statistics + db.session.execute(f""" + DELETE FROM receiver_statistics + WHERE date = '{date_str}'; + + INSERT INTO receiver_statistics AS rs (date, receiver_id, is_trustworthy, max_distance, max_normalized_quality, messages_count, coverages_count, senders_count) + SELECT + tmp.date, + tmp.receiver_id, + + is_trustworthy, + + MAX(tmp.max_distance) AS max_distance, + MAX(tmp.max_normalized_quality) AS max_normalized_quality, + SUM(tmp.messages_count) AS messages_count, + COUNT(DISTINCT tmp.location_mgrs_short) AS coverages_count, + COUNT(DISTINCT tmp.sender_id) AS senders_count + FROM coverage_statistics AS tmp + WHERE tmp.date = '{date_str}' + GROUP BY date, receiver_id, is_trustworthy; + """) + + db.session.commit() + + +@celery.task(name="update_sender_direction_statistics") +def update_sender_direction_statistics(): + """ Update sender_direction_statistics.""" + + db.session.execute(""" + DELETE FROM sender_direction_statistics; + + INSERT INTO sender_direction_statistics(sender_id, receiver_id, directions_count, messages_count, direction_data) + SELECT + sq2.sender_id, + sq2.receiver_id, + COUNT(sq2.*) AS directions_count, + SUM(sq2.messages_count) AS messages_count, + json_agg(json_build_object('direction', direction, 'messages_count', messages_count, 'max_range', max_range)) AS direction_data + FROM ( + SELECT + sq.sender_id, + sq.receiver_id, + sq.direction, + COUNT(sq.*) AS messages_count, + MAX(sq.max_range) AS max_range + FROM ( + SELECT + s.id AS sender_id, + r.id AS receiver_id, + 10000 * 10^(sp.normalized_quality/20.0) AS max_range, + CASE + WHEN sp.bearing-sp.track < 0 + THEN CAST((sp.bearing-sp.track+360)/10 AS INTEGER)*10 + ELSE CAST((sp.bearing-sp.track)/10 AS INTEGER)*10 + END AS direction + FROM sender_positions AS sp + INNER JOIN senders s ON sp.name = s.name + INNER JOIN receivers r ON sp.receiver_name = r.name + WHERE + sp.track IS NOT NULL AND sp.bearing IS NOT NULL AND sp.normalized_quality IS NOT NULL + AND sp.agl >= 200 + AND turn_rate BETWEEN -10.0 AND 10.0 + AND climb_rate BETWEEN -3.0 AND 3.0 + ) AS sq + GROUP BY sq.sender_id, sq.receiver_id, sq.direction + ORDER BY sq.sender_id, sq.receiver_id, sq.direction + ) AS sq2 + GROUP BY sq2.sender_id, sq2.receiver_id; + """) diff --git a/app/templates/airport_detail.html b/app/templates/airport_detail.html index 0530266..8a75800 100644 --- a/app/templates/airport_detail.html +++ b/app/templates/airport_detail.html @@ -21,21 +21,36 @@
-

Seen Devices

+

Receivers

+ + + + + + {% for receiver in airport.receivers %} + + + + {% endfor %} +
Name
{{ receiver.name }}
+
+ +
+

Seen Senders

- - - - + + + + - {% for device in devices %} + {% for sender in senders %} - - - + + + {% endfor %}
AddressRegistrationLast takeoff/landingSoftware versionNameLast takeoff/landingHardware versionSoftware version
{{ device.address }}{% if device.info is none %}-{% else %}{{ device.info.registration }}{% endif %}{% if device.takeoff_landings %}{% set last_action = device.takeoff_landings|last %}{% if last_action.is_takeoff == True %}↗{% else %}↘{% endif %} @ {{ last_action.timestamp.strftime('%Y-%m-%d %H:%M:%S') }}{% endif %} - {% if device.software_version is not none %}{{ device.software_version }}{% else %}-{% endif %}{{ sender|to_html_link|safe }}{% if sender.takeoff_landings %}{% set last_action = sender.takeoff_landings|last %}{% if last_action.is_takeoff == True %}↗{% else %}↘{% endif %} @ {{ last_action.timestamp.strftime('%Y-%m-%d %H:%M:%S') }}{% endif %} + {% if sender.hardware_version is not none %}{{ sender.hardware_version }}{% else %}-{% endif %}{% if sender.software_version is not none %}{{ sender.software_version }}{% else %}-{% endif %}
diff --git a/app/templates/airports.html b/app/templates/airports.html index 4af1157..939c535 100644 --- a/app/templates/airports.html +++ b/app/templates/airports.html @@ -23,6 +23,7 @@ + @@ -30,8 +31,9 @@ {% for airport in airports %} - + + + {% endfor %}
#Country Name Logbook (takeoff and landings)
{{ loop.index }} - {{ sel_country }} {{ airport.name }}Logbook{{ sel_country }}{{ airport|to_html_link|safe }}Logbook
diff --git a/app/templates/base.html b/app/templates/base.html index 70d659f..fa67db5 100644 --- a/app/templates/base.html +++ b/app/templates/base.html @@ -5,7 +5,7 @@ {% endblock %} {% block navbar %} - +