From d5e3e63d6df046f8b1c9a82228df531aff1bc602 Mon Sep 17 00:00:00 2001 From: geeksville Date: Sun, 20 Sep 2020 13:04:29 -0700 Subject: [PATCH 01/10] begin mqtt planning --- docs/software/mqtt.md | 130 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 docs/software/mqtt.md diff --git a/docs/software/mqtt.md b/docs/software/mqtt.md new file mode 100644 index 00000000..30ea0c42 --- /dev/null +++ b/docs/software/mqtt.md @@ -0,0 +1,130 @@ +# MQTT / remote attributes / on-device app API + +This is a mini-doc/RFC sketching out a development plan to satisfy a number of 1.1 goals. + +## Short term goals + +- We want a clean API for novice developers to write mini "apps" that run **on the device** with the existing messaging/location "apps". +- We want the ability to have a gateway web service, so that if any node in the mesh can connect to the internet (via its connected phone app or directly) then that node will provide bidirectional messaging between nodes and the internet. +- We want an easy way for novice developers to remotely read and control GPIOs (because this is an often requested use case), without those developers having to write any device code. +- We want a way to gateway text messaging between our current private meshes and the broader internet (when that mesh is able to connect to the internet) +- We want a way to remotely set any device/channel parameter on a node. This is particularly important for administering physically inaccessible router nodes. Ideally this mechanism would also be used for administering the local node (so one common mechanism for both cases). +- This work should be independent of our current (semi-custom) LoRa transport, so that in the future we can swap out that transport if we wish (to QMesh or Reticulum?) +- Our networks are (usually) very slow and low bandwidth, so the messaging must be very airtime efficient. + +## Long term goals + +- Store and forward messaging should be supported, so apps can send messages that might be delivered to their destination in **hours** or **days** if a node/mesh was partitioned. + +## Security + +Mini-apps API can bind to particular channels. They will only see messages sent on that channel. + +During the 1.1 timeframe only one channel is supported per node, but eventually we will do things like "remote admin operations / channel settings etc..." are on the "Control" channel and only especially trusted users should have the keys to access that channel. This means that during 1.1 you should assume that **any** user you grant access to your mesh (if technically knowledgeable enough) could change network settings. So you should still think of your meshes as private tools for friends. FIXME - how would this work with remote mqtt? + +## Efficient MQTT + +A gateway-device will contact the MQTT broker. For each operation it will use the meshtastic node ID as the MQTT "client ID". + +### Topics + +A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/) heirarchy for a complete functioning mesh: + +| Topic | Description | +| --------------------------------------- | --------------------------------------------------- | +| MESHID/NODEID/id/upd | A node ID update broadcast | +| MESHID/NODEID/pos/upd | A position update broadcast | +| MESHID/NODEID/pos/req | A position update request | +| MESHID/USERID/msg/text/DESTCLASS/DESTID | A text message from USERID to DESTCLASS/DESTID | +| MESHID/NODEID/msg/bin/DESTCLASS/DESTID | A binary message from NODEID to DESTCLASS/DESTCLASS | +| MESHID/NODEID/gpio/set/GPIONUM | Set a GPIO output | +| MESHID/NODEID/gpio/get/GPIONUM | Try to read a GPIO | +| MESHID/NODEID/gpio/upd/GPIONUM | Contains the read GPIO value | +| MESHID/NODEID/attr/ATTRNAME/req | Request a current attribute value | +| MESHID/NODEID/attr/ATTRNAME/set | Set an attribute value | +| MESHID/NODEID/app/APPNUM/# | An topic from an unregistered/unknown app | + +Gateway nodes (via code running in the phone) will contain two tables to whitelist particular traffic to either be delivered toward the internet, or down toward the mesh. Users that are developing custom apps will be able to customize these filters/subscriptions. + +#### Default ToInternet filters + +These filters are used to whitelist particular traffic - only traffic that matches a filter will be forwarded to the internet MQTT broker. + +| Pattern | Description | +| ---------------- | -------------------------------------------------------------------------- | +| +/+/id/# | Only if set for 'no privacy' | +| +/+/pos/upd | Only if set for 'no privacy' - useful for showing all nodes on a world map | +| +/+/msg/text/W/+ | For internet messaging feature | + +#### Default FromInternet subscriptions + +The gateway node will always subscribe to certain topics on the broker so that it can forward those topics into the mesh. + +| Pattern | Description | +| --------------------- | ------------------------------------------------------------------------------- | +| MESHID/+/msg/text/W/+ | To receive text messages from the internet (where the sender knows our mesh ID) | +| +/+/msg/text/W/USERID | For each named user on the local mesh, to receive messages bound for that user | + +The provided example MQTT broker from Geeksville will also have filters: + +#### MESHID + +#### NODEID + +#### USERID + +#### DESTCLASS + +Is used to filter whole classes of destination IDs (DESTID). Can be... + +- L - Local, for this mesh only. +- W - World, for this mesh and the broader internet + +#### DESTID + +Can be... + +- an internet username: kevinh@geeksville.com +- ^ALL for anyone +- An app ID (to allow apps out in the web to receive arbitrary binary data from nodes or simply other apps using meshtastic as a transport). They would connect to the MQTT broker and subscribe to their topic + +### Named attribute API + +### Name to ID mapping + +MQTT topic strings are very long and potentially expensive over the slow LORA network. Also, we don't want to burden each (dumb) node in the mesh with having to regex match against them. For that reason, well known topics map to (small) "topic IDs". For portions of the topic that correspond to a wildcard, those strings are provided as "topic arguments". This means that only the phone app needs to consider full ... FIXME. + +## Work items + +### Cleanup/refactoring of existing codebase + +- Refactor the position features into a position "mini-app". Use only the new public on-device API to implement this app. +- Refactor the on device texting features into a messaging "mini-app". (Similar to the position mini-app) + +### New 'no-code-IOT' mini-app + +Add a new 'remote GPIO/serial port/SPI/I2C access' mini-app. This new standard app would use the MQTT messaging layer to let users (devs that don't need to write device code) do basic (potentially dangerous) operations remotely. + +#### Supported operations in the initial release + +Initially supported features for no-code-IOT. + +- Set any GPIO +- Read any GPIO + +#### Supported operations eventually + +General ideas for no-code IOT. + +- Subscribe for notification of GPIO input status change (i.e. when pin goes low, send my app a message) +- Write/read N bytes over I2C/SPI bus Y (as one atomic I2C/SPI transaction) +- Send N bytes out serial port Z +- Subscribe for notification for when regex X matches the bytes that were received on serial port Z + +### Later release features (1.2) + +- Allow radios to be on multiple channels at once. Each channel will have its own encryption keys. + +``` + +``` From bc22ab7b87a04c54d8f067e21bd99a8f788f24c3 Mon Sep 17 00:00:00 2001 From: geeksville Date: Sun, 20 Sep 2020 13:07:53 -0700 Subject: [PATCH 02/10] riot.im --- docs/software/mqtt.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/software/mqtt.md b/docs/software/mqtt.md index 30ea0c42..b5a165fc 100644 --- a/docs/software/mqtt.md +++ b/docs/software/mqtt.md @@ -101,6 +101,10 @@ MQTT topic strings are very long and potentially expensive over the slow LORA ne - Refactor the position features into a position "mini-app". Use only the new public on-device API to implement this app. - Refactor the on device texting features into a messaging "mini-app". (Similar to the position mini-app) +### Riot.im bridge + +There is apparently already a riot.im bridge for MQTT. That will possibly need to be customized a bit. But by doing this, we should be able to let random riot.im users send/receive messages to/from any meshtastic device. (FIXME add link and ponder security) + ### New 'no-code-IOT' mini-app Add a new 'remote GPIO/serial port/SPI/I2C access' mini-app. This new standard app would use the MQTT messaging layer to let users (devs that don't need to write device code) do basic (potentially dangerous) operations remotely. From 26d50fda9aed5f9063ac200f4d4b2401d826dbe8 Mon Sep 17 00:00:00 2001 From: geeksville Date: Sun, 20 Sep 2020 18:08:10 -0700 Subject: [PATCH 03/10] Update mqtt doc --- docs/software/mqtt.md | 166 +++++++++++++++++++++++++++++++----------- 1 file changed, 124 insertions(+), 42 deletions(-) diff --git a/docs/software/mqtt.md b/docs/software/mqtt.md index b5a165fc..4ec96484 100644 --- a/docs/software/mqtt.md +++ b/docs/software/mqtt.md @@ -1,8 +1,43 @@ -# MQTT / remote attributes / on-device app API + +# 1. Table of Contents +- [1. Table of Contents](#1-table-of-contents) + - [1.1. Abstract](#11-abstract) + - [1.2. Short term goals](#12-short-term-goals) + - [1.3. Long term goals](#13-long-term-goals) + - [1.4. Security](#14-security) + - [1.5. Efficient MQTT](#15-efficient-mqtt) + - [1.5.1. Topics](#151-topics) + - [1.5.1.1. MESHID](#1511-meshid) + - [1.5.1.2. NODEID](#1512-nodeid) + - [1.5.1.3. DESTCLASS](#1513-destclass) + - [1.5.1.4. DESTID](#1514-destid) + - [1.5.1.5. USERID](#1515-userid) + - [1.5.2. Gateway nodes](#152-gateway-nodes) + - [1.5.2.1. Default ToInternet filters](#1521-default-tointernet-filters) + - [1.5.2.2. Default FromInternet subscriptions](#1522-default-frominternet-subscriptions) + - [1.5.3. Optional web services](#153-optional-web-services) + - [1.5.3.1. Public MQTT broker service](#1531-public-mqtt-broker-service) + - [1.5.3.2. Riot.im messaging bridge](#1532-riotim-messaging-bridge) + - [1.5.4. Named attribute API](#154-named-attribute-api) + - [1.5.5. Name to ID mapping](#155-name-to-id-mapping) + - [1.6. Development plan](#16-development-plan) + - [1.6.1. Cleanup/refactoring of existing codebase](#161-cleanuprefactoring-of-existing-codebase) + - [1.6.2. New 'no-code-IOT' mini-app](#162-new-no-code-iot-mini-app) + - [1.6.2.1. Supported operations in the initial release](#1621-supported-operations-in-the-initial-release) + - [1.6.2.2. Supported operations eventually](#1622-supported-operations-eventually) + - [1.6.3. Later release features (1.2)](#163-later-release-features-12) + +## 1.1. Abstract This is a mini-doc/RFC sketching out a development plan to satisfy a number of 1.1 goals. -## Short term goals +- [MQTT](https://opensource.com/article/18/6/mqtt) internet accessible API. Issue #[369](https://github.com/meshtastic/Meshtastic-device/issues/169) +- An open API to easily run custom mini-apps on the devices +- A text messaging bridge when a node in the mesh can gateway to the internet. Issue #[353](https://github.com/meshtastic/Meshtastic-device/issues/353) +- An easy way to let desktop app developers remotely control GPIOs. Issue #[182](https://github.com/meshtastic/Meshtastic-device/issues/182) +- Remote attribute access (to change settings of distant nodes). Issue #182 + +## 1.2. Short term goals - We want a clean API for novice developers to write mini "apps" that run **on the device** with the existing messaging/location "apps". - We want the ability to have a gateway web service, so that if any node in the mesh can connect to the internet (via its connected phone app or directly) then that node will provide bidirectional messaging between nodes and the internet. @@ -12,23 +47,25 @@ This is a mini-doc/RFC sketching out a development plan to satisfy a number of 1 - This work should be independent of our current (semi-custom) LoRa transport, so that in the future we can swap out that transport if we wish (to QMesh or Reticulum?) - Our networks are (usually) very slow and low bandwidth, so the messaging must be very airtime efficient. -## Long term goals +## 1.3. Long term goals - Store and forward messaging should be supported, so apps can send messages that might be delivered to their destination in **hours** or **days** if a node/mesh was partitioned. -## Security +## 1.4. Security Mini-apps API can bind to particular channels. They will only see messages sent on that channel. -During the 1.1 timeframe only one channel is supported per node, but eventually we will do things like "remote admin operations / channel settings etc..." are on the "Control" channel and only especially trusted users should have the keys to access that channel. This means that during 1.1 you should assume that **any** user you grant access to your mesh (if technically knowledgeable enough) could change network settings. So you should still think of your meshes as private tools for friends. FIXME - how would this work with remote mqtt? +During the 1.1 timeframe only one channel is supported per node, but eventually we will do things like "remote admin operations / channel settings etc..." are on the "Control" channel and only especially trusted users should have the keys to access that channel. -## Efficient MQTT +See "Named Attribute API" section for special access control to prevent remote access to device settings. -A gateway-device will contact the MQTT broker. For each operation it will use the meshtastic node ID as the MQTT "client ID". +## 1.5. Efficient MQTT -### Topics +A gateway-device will contact the MQTT broker. For each operation it will use the meshtastic "MESHID/NODEID" tuple as the MQTT "client ID". MESHIDs will be (TBD somehow) tracked and authenticated out-of-band. -A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/) heirarchy for a complete functioning mesh: +### 1.5.1. Topics + +A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/) hierarchy for a complete functioning mesh: | Topic | Description | | --------------------------------------- | --------------------------------------------------- | @@ -44,19 +81,56 @@ A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics- | MESHID/NODEID/attr/ATTRNAME/set | Set an attribute value | | MESHID/NODEID/app/APPNUM/# | An topic from an unregistered/unknown app | +#### 1.5.1.1. MESHID + +A unique ID for this mesh. There will be some sort of key exchange process so that the mesh ID can not be impersonated by other meshes. + +#### 1.5.1.2. NODEID + +The unique ID for a node. A hex string that starts with a ! symbol. + +#### 1.5.1.3. DESTCLASS + +The type of DESTID this message should be delivered to. A short one letter sequence: + +| Symbol | Description | +| ------ | ------------------------------------------------------------- | +| R | riot.im | +| L | local mesh node ID or ^all | +| A | an application specific message, ID will be an APP ID | +| S | SMS gateway, DESTID is a phone number to reach via Twilio.com | +| E | Emergency message, see bug #fixme for more context | + +#### 1.5.1.4. DESTID + +Can be... + +- an internet username: kevinh@geeksville.com +- ^ALL for anyone +- An app ID (to allow apps out in the web to receive arbitrary binary data from nodes or simply other apps using meshtastic as a transport). They would connect to the MQTT broker and subscribe to their topic + +#### 1.5.1.5. USERID + +A user ID string. This string is either a user ID if known or a nodeid to simply deliver the message to whoever the local user is of a particular device (i.e. person who might see the screen). FIXME, see what riot.im uses and perhaps use that convention? Or use the signal +phone number convention? Or the email addr? + +### 1.5.2. Gateway nodes + +Any meshtastic node that has a direct connection to the internet (either via a helper app or installed wifi/4G/satellite hardware) can function as a "Gateway node". + Gateway nodes (via code running in the phone) will contain two tables to whitelist particular traffic to either be delivered toward the internet, or down toward the mesh. Users that are developing custom apps will be able to customize these filters/subscriptions. -#### Default ToInternet filters +#### 1.5.2.1. Default ToInternet filters These filters are used to whitelist particular traffic - only traffic that matches a filter will be forwarded to the internet MQTT broker. -| Pattern | Description | -| ---------------- | -------------------------------------------------------------------------- | -| +/+/id/# | Only if set for 'no privacy' | -| +/+/pos/upd | Only if set for 'no privacy' - useful for showing all nodes on a world map | -| +/+/msg/text/W/+ | For internet messaging feature | +| Pattern | Description | +| ---------------- | -------------------------------------------------------------------------------------------------------------------------- | +| +/+/id/# | Only if set for 'no privacy' | +| +/+/pos/upd | Only if set for 'no privacy' - useful for showing all nodes on a world map | +| +/+/msg/text/W/+ | For internet messaging feature | +| +/+/app/APPNUM/# | Only if "send app APPNUM" has been set in gateway settings - for app developers who want their traffic routed to the world | -#### Default FromInternet subscriptions +#### 1.5.2.2. Default FromInternet subscriptions The gateway node will always subscribe to certain topics on the broker so that it can forward those topics into the mesh. @@ -65,58 +139,68 @@ The gateway node will always subscribe to certain topics on the broker so that i | MESHID/+/msg/text/W/+ | To receive text messages from the internet (where the sender knows our mesh ID) | | +/+/msg/text/W/USERID | For each named user on the local mesh, to receive messages bound for that user | -The provided example MQTT broker from Geeksville will also have filters: +### 1.5.3. Optional web services -#### MESHID +#### 1.5.3.1. Public MQTT broker service -#### NODEID +@Geeksville will provide a standard [MQTT broker](https://mosquitto.org/) on the web to facilitate use of this service, but clients can use any MQTT broker they choose. Geeksville will initially not charge for the use of this broker, but if it becomes a burden he might ask for donations or require a payment for the use of the server. -#### USERID +The provided public MQTT broker from geeksville.com will also have filters to ensure: -#### DESTCLASS +- only authenticated MESHIDs can publish under that ID +- messages sent/to from the riot.im text message bridge can only be seen by that bridge or the intended destination/source mesh Is used to filter whole classes of destination IDs (DESTID). Can be... - L - Local, for this mesh only. - W - World, for this mesh and the broader internet -#### DESTID +#### 1.5.3.2. Riot.im messaging bridge -Can be... +@Geeksville will run a riot.im bridge that talks to the public MQTT broker and sends/receives into the riot.im network. -- an internet username: kevinh@geeksville.com -- ^ALL for anyone -- An app ID (to allow apps out in the web to receive arbitrary binary data from nodes or simply other apps using meshtastic as a transport). They would connect to the MQTT broker and subscribe to their topic +There is apparently already a riot.im [bridge](https://matrix.org/bridges/) for MQTT. That will possibly need to be customized a bit. But by doing this, we should be able to let random riot.im users send/receive messages to/from any meshtastic device. (FIXME add link and ponder security) -### Named attribute API +### 1.5.4. Named attribute API -### Name to ID mapping +The current channel and node settings are set/read using a special protobuf exchange between a local client and the meshtastic device. In version 1.1 that mechanism will be changed so that settings are set/read using MQTT (even when done locally). This will enable remote node adminstration (even conceivably through the internet). -MQTT topic strings are very long and potentially expensive over the slow LORA network. Also, we don't want to burden each (dumb) node in the mesh with having to regex match against them. For that reason, well known topics map to (small) "topic IDs". For portions of the topic that correspond to a wildcard, those strings are provided as "topic arguments". This means that only the phone app needs to consider full ... FIXME. +To provide some basic security a new named attribute name "seckey" can be set. If set, any attribute operations must include that get with their operation request. Note: This mechanism still assumes that users you grant permission to access your local mesh are not 'adversaries'. A technically competent user could discover the remote attribute key needed for attribute reading/writing. In the 1.2ish timeframe we will add the concept of multiple channels and in that case, remote attribute operations will be on their own secured channel that regular 'users' can not see. -## Work items +### 1.5.5. Name to ID mapping -### Cleanup/refactoring of existing codebase +MQTT topic strings are very long and potentially expensive over the slow LORA network. Also, we don't want to burden each (dumb) node in the mesh with having to regex match against them. For that reason, well known topics map to (small) "topic IDs". For portions of the topic that correspond to a wildcard, those strings are provided as "topic arguments". This means that only the phone app needs to consider full topic strings. Device nodes will only understand integer topic IDs and their arguments. +FIXME, add more details to this section and figure out how unassigned apps/topics work in this framework. + +## 1.6. Development plan + +Given the previous problem/goals statement, here's the initial thoughts on the work items required. As this idea becomes a bit more fully baked we should add details +on how this will be implemented and guesses at approximate work items. + +### 1.6.1. Cleanup/refactoring of existing codebase + +- Change nodeIDs to be base64 instead of eight hex digits. +- Add the concept of topic IDs and topic arguments to the protobufs and the device code. - Refactor the position features into a position "mini-app". Use only the new public on-device API to implement this app. - Refactor the on device texting features into a messaging "mini-app". (Similar to the position mini-app) +- Add first cut of the "gateway node" code (i.e. MQTT broker client) to the python API (very little code needed for this component) +- Confirm that texting works to/from the internet +- Confirm that positions are optionally sent to the internet +- Add the first cut of the "gateway node" code to the android app (very little code needed for this component) -### Riot.im bridge +### 1.6.2. New 'no-code-IOT' mini-app -There is apparently already a riot.im bridge for MQTT. That will possibly need to be customized a bit. But by doing this, we should be able to let random riot.im users send/receive messages to/from any meshtastic device. (FIXME add link and ponder security) +Add a new 'remote GPIO/serial port/SPI/I2C access' mini-app. This new standard app would use the MQTT messaging layer to let users (developers that don't need to write device code) do basic (potentially dangerous) operations remotely. -### New 'no-code-IOT' mini-app - -Add a new 'remote GPIO/serial port/SPI/I2C access' mini-app. This new standard app would use the MQTT messaging layer to let users (devs that don't need to write device code) do basic (potentially dangerous) operations remotely. - -#### Supported operations in the initial release +#### 1.6.2.1. Supported operations in the initial release Initially supported features for no-code-IOT. - Set any GPIO - Read any GPIO -#### Supported operations eventually +#### 1.6.2.2. Supported operations eventually General ideas for no-code IOT. @@ -125,10 +209,8 @@ General ideas for no-code IOT. - Send N bytes out serial port Z - Subscribe for notification for when regex X matches the bytes that were received on serial port Z -### Later release features (1.2) +### 1.6.3. Later release features (1.2) - Allow radios to be on multiple channels at once. Each channel will have its own encryption keys. -``` -``` From 0ba4925f75548078791f0bcd0e895d2752fcbce9 Mon Sep 17 00:00:00 2001 From: geeksville Date: Mon, 21 Sep 2020 11:54:54 -0700 Subject: [PATCH 04/10] add @mc-hamster comment --- docs/software/mqtt.md | 89 +++++++++++++++++++++++-------------------- 1 file changed, 47 insertions(+), 42 deletions(-) diff --git a/docs/software/mqtt.md b/docs/software/mqtt.md index 4ec96484..9b44af76 100644 --- a/docs/software/mqtt.md +++ b/docs/software/mqtt.md @@ -5,27 +5,28 @@ - [1.2. Short term goals](#12-short-term-goals) - [1.3. Long term goals](#13-long-term-goals) - [1.4. Security](#14-security) - - [1.5. Efficient MQTT](#15-efficient-mqtt) - - [1.5.1. Topics](#151-topics) - - [1.5.1.1. MESHID](#1511-meshid) - - [1.5.1.2. NODEID](#1512-nodeid) - - [1.5.1.3. DESTCLASS](#1513-destclass) - - [1.5.1.4. DESTID](#1514-destid) - - [1.5.1.5. USERID](#1515-userid) - - [1.5.2. Gateway nodes](#152-gateway-nodes) - - [1.5.2.1. Default ToInternet filters](#1521-default-tointernet-filters) - - [1.5.2.2. Default FromInternet subscriptions](#1522-default-frominternet-subscriptions) - - [1.5.3. Optional web services](#153-optional-web-services) - - [1.5.3.1. Public MQTT broker service](#1531-public-mqtt-broker-service) - - [1.5.3.2. Riot.im messaging bridge](#1532-riotim-messaging-bridge) - - [1.5.4. Named attribute API](#154-named-attribute-api) - - [1.5.5. Name to ID mapping](#155-name-to-id-mapping) - - [1.6. Development plan](#16-development-plan) - - [1.6.1. Cleanup/refactoring of existing codebase](#161-cleanuprefactoring-of-existing-codebase) - - [1.6.2. New 'no-code-IOT' mini-app](#162-new-no-code-iot-mini-app) - - [1.6.2.1. Supported operations in the initial release](#1621-supported-operations-in-the-initial-release) - - [1.6.2.2. Supported operations eventually](#1622-supported-operations-eventually) - - [1.6.3. Later release features (1.2)](#163-later-release-features-12) + - [1.5. On device API](#15-on-device-api) + - [1.6. Efficient MQTT](#16-efficient-mqtt) + - [1.6.1. Topics](#161-topics) + - [1.6.1.1. MESHID](#1611-meshid) + - [1.6.1.2. NODEID](#1612-nodeid) + - [1.6.1.3. DESTCLASS](#1613-destclass) + - [1.6.1.4. DESTID](#1614-destid) + - [1.6.1.5. USERID](#1615-userid) + - [1.6.2. Gateway nodes](#162-gateway-nodes) + - [1.6.2.1. Default ToInternet filters](#1621-default-tointernet-filters) + - [1.6.2.2. Default FromInternet subscriptions](#1622-default-frominternet-subscriptions) + - [1.6.3. Optional web services](#163-optional-web-services) + - [1.6.3.1. Public MQTT broker service](#1631-public-mqtt-broker-service) + - [1.6.3.2. Riot.im messaging bridge](#1632-riotim-messaging-bridge) + - [1.6.4. Named attribute API](#164-named-attribute-api) + - [1.6.5. Name to ID mapping](#165-name-to-id-mapping) + - [1.7. Development plan](#17-development-plan) + - [1.7.1. Cleanup/refactoring of existing codebase](#171-cleanuprefactoring-of-existing-codebase) + - [1.7.2. New 'no-code-IOT' mini-app](#172-new-no-code-iot-mini-app) + - [1.7.2.1. Supported operations in the initial release](#1721-supported-operations-in-the-initial-release) + - [1.7.2.2. Supported operations eventually](#1722-supported-operations-eventually) + - [1.7.3. Later release features (1.2)](#173-later-release-features-12) ## 1.1. Abstract @@ -59,11 +60,15 @@ During the 1.1 timeframe only one channel is supported per node, but eventually See "Named Attribute API" section for special access control to prevent remote access to device settings. -## 1.5. Efficient MQTT +## 1.5. On device API + +FIXME - add an example of the on-device API. Possibly by showing the new position or texting code. + +## 1.6. Efficient MQTT A gateway-device will contact the MQTT broker. For each operation it will use the meshtastic "MESHID/NODEID" tuple as the MQTT "client ID". MESHIDs will be (TBD somehow) tracked and authenticated out-of-band. -### 1.5.1. Topics +### 1.6.1. Topics A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/) hierarchy for a complete functioning mesh: @@ -81,15 +86,15 @@ A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics- | MESHID/NODEID/attr/ATTRNAME/set | Set an attribute value | | MESHID/NODEID/app/APPNUM/# | An topic from an unregistered/unknown app | -#### 1.5.1.1. MESHID +#### 1.6.1.1. MESHID A unique ID for this mesh. There will be some sort of key exchange process so that the mesh ID can not be impersonated by other meshes. -#### 1.5.1.2. NODEID +#### 1.6.1.2. NODEID The unique ID for a node. A hex string that starts with a ! symbol. -#### 1.5.1.3. DESTCLASS +#### 1.6.1.3. DESTCLASS The type of DESTID this message should be delivered to. A short one letter sequence: @@ -101,7 +106,7 @@ The type of DESTID this message should be delivered to. A short one letter seque | S | SMS gateway, DESTID is a phone number to reach via Twilio.com | | E | Emergency message, see bug #fixme for more context | -#### 1.5.1.4. DESTID +#### 1.6.1.4. DESTID Can be... @@ -109,17 +114,17 @@ Can be... - ^ALL for anyone - An app ID (to allow apps out in the web to receive arbitrary binary data from nodes or simply other apps using meshtastic as a transport). They would connect to the MQTT broker and subscribe to their topic -#### 1.5.1.5. USERID +#### 1.6.1.5. USERID A user ID string. This string is either a user ID if known or a nodeid to simply deliver the message to whoever the local user is of a particular device (i.e. person who might see the screen). FIXME, see what riot.im uses and perhaps use that convention? Or use the signal +phone number convention? Or the email addr? -### 1.5.2. Gateway nodes +### 1.6.2. Gateway nodes Any meshtastic node that has a direct connection to the internet (either via a helper app or installed wifi/4G/satellite hardware) can function as a "Gateway node". Gateway nodes (via code running in the phone) will contain two tables to whitelist particular traffic to either be delivered toward the internet, or down toward the mesh. Users that are developing custom apps will be able to customize these filters/subscriptions. -#### 1.5.2.1. Default ToInternet filters +#### 1.6.2.1. Default ToInternet filters These filters are used to whitelist particular traffic - only traffic that matches a filter will be forwarded to the internet MQTT broker. @@ -130,7 +135,7 @@ These filters are used to whitelist particular traffic - only traffic that match | +/+/msg/text/W/+ | For internet messaging feature | | +/+/app/APPNUM/# | Only if "send app APPNUM" has been set in gateway settings - for app developers who want their traffic routed to the world | -#### 1.5.2.2. Default FromInternet subscriptions +#### 1.6.2.2. Default FromInternet subscriptions The gateway node will always subscribe to certain topics on the broker so that it can forward those topics into the mesh. @@ -139,9 +144,9 @@ The gateway node will always subscribe to certain topics on the broker so that i | MESHID/+/msg/text/W/+ | To receive text messages from the internet (where the sender knows our mesh ID) | | +/+/msg/text/W/USERID | For each named user on the local mesh, to receive messages bound for that user | -### 1.5.3. Optional web services +### 1.6.3. Optional web services -#### 1.5.3.1. Public MQTT broker service +#### 1.6.3.1. Public MQTT broker service @Geeksville will provide a standard [MQTT broker](https://mosquitto.org/) on the web to facilitate use of this service, but clients can use any MQTT broker they choose. Geeksville will initially not charge for the use of this broker, but if it becomes a burden he might ask for donations or require a payment for the use of the server. @@ -155,30 +160,30 @@ Is used to filter whole classes of destination IDs (DESTID). Can be... - L - Local, for this mesh only. - W - World, for this mesh and the broader internet -#### 1.5.3.2. Riot.im messaging bridge +#### 1.6.3.2. Riot.im messaging bridge @Geeksville will run a riot.im bridge that talks to the public MQTT broker and sends/receives into the riot.im network. There is apparently already a riot.im [bridge](https://matrix.org/bridges/) for MQTT. That will possibly need to be customized a bit. But by doing this, we should be able to let random riot.im users send/receive messages to/from any meshtastic device. (FIXME add link and ponder security) -### 1.5.4. Named attribute API +### 1.6.4. Named attribute API The current channel and node settings are set/read using a special protobuf exchange between a local client and the meshtastic device. In version 1.1 that mechanism will be changed so that settings are set/read using MQTT (even when done locally). This will enable remote node adminstration (even conceivably through the internet). To provide some basic security a new named attribute name "seckey" can be set. If set, any attribute operations must include that get with their operation request. Note: This mechanism still assumes that users you grant permission to access your local mesh are not 'adversaries'. A technically competent user could discover the remote attribute key needed for attribute reading/writing. In the 1.2ish timeframe we will add the concept of multiple channels and in that case, remote attribute operations will be on their own secured channel that regular 'users' can not see. -### 1.5.5. Name to ID mapping +### 1.6.5. Name to ID mapping MQTT topic strings are very long and potentially expensive over the slow LORA network. Also, we don't want to burden each (dumb) node in the mesh with having to regex match against them. For that reason, well known topics map to (small) "topic IDs". For portions of the topic that correspond to a wildcard, those strings are provided as "topic arguments". This means that only the phone app needs to consider full topic strings. Device nodes will only understand integer topic IDs and their arguments. FIXME, add more details to this section and figure out how unassigned apps/topics work in this framework. -## 1.6. Development plan +## 1.7. Development plan Given the previous problem/goals statement, here's the initial thoughts on the work items required. As this idea becomes a bit more fully baked we should add details on how this will be implemented and guesses at approximate work items. -### 1.6.1. Cleanup/refactoring of existing codebase +### 1.7.1. Cleanup/refactoring of existing codebase - Change nodeIDs to be base64 instead of eight hex digits. - Add the concept of topic IDs and topic arguments to the protobufs and the device code. @@ -189,18 +194,18 @@ on how this will be implemented and guesses at approximate work items. - Confirm that positions are optionally sent to the internet - Add the first cut of the "gateway node" code to the android app (very little code needed for this component) -### 1.6.2. New 'no-code-IOT' mini-app +### 1.7.2. New 'no-code-IOT' mini-app Add a new 'remote GPIO/serial port/SPI/I2C access' mini-app. This new standard app would use the MQTT messaging layer to let users (developers that don't need to write device code) do basic (potentially dangerous) operations remotely. -#### 1.6.2.1. Supported operations in the initial release +#### 1.7.2.1. Supported operations in the initial release Initially supported features for no-code-IOT. - Set any GPIO - Read any GPIO -#### 1.6.2.2. Supported operations eventually +#### 1.7.2.2. Supported operations eventually General ideas for no-code IOT. @@ -209,7 +214,7 @@ General ideas for no-code IOT. - Send N bytes out serial port Z - Subscribe for notification for when regex X matches the bytes that were received on serial port Z -### 1.6.3. Later release features (1.2) +### 1.7.3. Later release features (1.2) - Allow radios to be on multiple channels at once. Each channel will have its own encryption keys. From 8bdbbfbe16186a8baff8e00398ab708ad070cefb Mon Sep 17 00:00:00 2001 From: geeksville Date: Mon, 21 Sep 2020 12:44:30 -0700 Subject: [PATCH 05/10] add notes about tuple --- docs/software/mqtt.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/software/mqtt.md b/docs/software/mqtt.md index 9b44af76..0d3f7745 100644 --- a/docs/software/mqtt.md +++ b/docs/software/mqtt.md @@ -34,7 +34,7 @@ This is a mini-doc/RFC sketching out a development plan to satisfy a number of 1 - [MQTT](https://opensource.com/article/18/6/mqtt) internet accessible API. Issue #[369](https://github.com/meshtastic/Meshtastic-device/issues/169) - An open API to easily run custom mini-apps on the devices -- A text messaging bridge when a node in the mesh can gateway to the internet. Issue #[353](https://github.com/meshtastic/Meshtastic-device/issues/353) +- A text messaging bridge when a node in the mesh can gateway to the internet. Issue #[353](https://github.com/meshtastic/Meshtastic-device/issues/353) and this nicely documented [android issue](https://github.com/meshtastic/Meshtastic-Android/issues/2). - An easy way to let desktop app developers remotely control GPIOs. Issue #[182](https://github.com/meshtastic/Meshtastic-device/issues/182) - Remote attribute access (to change settings of distant nodes). Issue #182 @@ -164,7 +164,7 @@ Is used to filter whole classes of destination IDs (DESTID). Can be... @Geeksville will run a riot.im bridge that talks to the public MQTT broker and sends/receives into the riot.im network. -There is apparently already a riot.im [bridge](https://matrix.org/bridges/) for MQTT. That will possibly need to be customized a bit. But by doing this, we should be able to let random riot.im users send/receive messages to/from any meshtastic device. (FIXME add link and ponder security) +There is apparently [already](https://github.com/derEisele/tuple) a riot.im [bridge](https://matrix.org/bridges/) for MQTT. That will possibly need to be customized a bit. But by doing this, we should be able to let random riot.im users send/receive messages to/from any meshtastic device. (FIXME ponder security). See this [issue](https://github.com/meshtastic/Meshtastic-Android/issues/2#issuecomment-645660990) with discussion with the dev. ### 1.6.4. Named attribute API From 8d04410f4582f5752cbf5d900ce2f66f26dd0d26 Mon Sep 17 00:00:00 2001 From: Kevin Hester Date: Sat, 26 Dec 2020 13:55:59 +0800 Subject: [PATCH 06/10] improve error descriptions --- src/error.h | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/error.h b/src/error.h index 9b50f670..b3926c9d 100644 --- a/src/error.h +++ b/src/error.h @@ -9,15 +9,23 @@ */ enum CriticalErrorCode { NoError = 0, + + /// A software bug was detected while trying to send lora packets ErrTxWatchdog = 1, + + /// A software bug was detected on entry to sleep ErrSleepEnterWait = 2, - // No Lora radio hardware could be found + /// No Lora radio hardware could be found ErrNoRadio = 3, + + /// Not normally used ErrUnspecified = 4, + + /// We failed while configuring a UBlox GPS ErrUBloxInitFailed = 5, - // This board was expected to have a power management chip and it is missing or broken + /// This board was expected to have a power management chip and it is missing or broken ErrNoAXP192 = 6 }; From 0c74303e9d567bf063271145ede7c4a1e78391f5 Mon Sep 17 00:00:00 2001 From: Kevin Hester Date: Sun, 27 Dec 2020 11:22:08 +0800 Subject: [PATCH 07/10] move criticalerror defs into .proto for cross platform support --- docs/software/TODO.md | 1 + docs/software/mqtt.md | 27 +++++++++++++- proto | 2 +- src/error.h | 29 +-------------- src/gps/UBloxGPS.cpp | 2 +- src/main.cpp | 4 +- src/mesh/NodeDB.cpp | 2 +- src/mesh/mesh.pb.c | 4 +- src/mesh/mesh.pb.h | 83 ++++++++++++++++++++++++++++++------------ src/mesh/portnums.pb.h | 8 ++-- src/sleep.cpp | 2 +- 11 files changed, 101 insertions(+), 63 deletions(-) diff --git a/docs/software/TODO.md b/docs/software/TODO.md index 6c338c81..69f22591 100644 --- a/docs/software/TODO.md +++ b/docs/software/TODO.md @@ -4,6 +4,7 @@ You probably don't care about this section - skip to the next one. For app cleanup: +* use structured logging to kep logs in ram. Also send logs as packets to api clients * DONE writeup nice python options docs (common cases, link to protobuf docs) * have android app link to user manual * DONE only do wantReplies once per packet type, if we change network settings force it again diff --git a/docs/software/mqtt.md b/docs/software/mqtt.md index 0d3f7745..06ec7bb0 100644 --- a/docs/software/mqtt.md +++ b/docs/software/mqtt.md @@ -30,6 +30,17 @@ ## 1.1. Abstract +FIXME: + +- add channels as security + - have a uplinkPolicy enum (none, up only, down only, updown, stay encrypted) +- add simpler mapping of channels/nodes/messages/portnums to mqtt topics +- leave payloads as raw packets/protobufs +- explain why not UDP + - need to have a server anyways so that nodes can reach each other from anywhere + - raw UDP is dropped **very** agressively by many cellular providers + - mqtt provides a nice/documented/standard security model to build upon + This is a mini-doc/RFC sketching out a development plan to satisfy a number of 1.1 goals. - [MQTT](https://opensource.com/article/18/6/mqtt) internet accessible API. Issue #[369](https://github.com/meshtastic/Meshtastic-device/issues/169) @@ -80,12 +91,24 @@ A sample [topic](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics- | MESHID/USERID/msg/text/DESTCLASS/DESTID | A text message from USERID to DESTCLASS/DESTID | | MESHID/NODEID/msg/bin/DESTCLASS/DESTID | A binary message from NODEID to DESTCLASS/DESTCLASS | | MESHID/NODEID/gpio/set/GPIONUM | Set a GPIO output | -| MESHID/NODEID/gpio/get/GPIONUM | Try to read a GPIO | -| MESHID/NODEID/gpio/upd/GPIONUM | Contains the read GPIO value | | MESHID/NODEID/attr/ATTRNAME/req | Request a current attribute value | | MESHID/NODEID/attr/ATTRNAME/set | Set an attribute value | | MESHID/NODEID/app/APPNUM/# | An topic from an unregistered/unknown app | +for encrypted packets (consumer would need to have access to the specified channel to be able to parse) + +encrypted/CHANNELID/MESHID/NODEID/PORTID + +If the channelid 'well known'/public it can be decrypted by a web service, in which case it will be decrypted by a web service and appear at: + +clear/MESHID/NODEID/PORTID + +FIXME, the payload published on the topic, will include the message, and full information about arrival time, who forwarded it, source channel etc... + +FIXME, figure out how channelids work +FIXME, figure out rules for store and forward + + #### 1.6.1.1. MESHID A unique ID for this mesh. There will be some sort of key exchange process so that the mesh ID can not be impersonated by other meshes. diff --git a/proto b/proto index 020ef9ee..9a7d8a03 160000 --- a/proto +++ b/proto @@ -1 +1 @@ -Subproject commit 020ef9eea8129756a0b45be5a3900b0355be4451 +Subproject commit 9a7d8a03cb051eb42787d4a06836b109677d8ef1 diff --git a/src/error.h b/src/error.h index b3926c9d..e290c403 100644 --- a/src/error.h +++ b/src/error.h @@ -2,32 +2,7 @@ #include -/** Error codes for critical errors - * - * The device might report these fault codes on the screen. If you encounter a fault code, please - * post on the meshtastic.discourse.group and we'll try to help. - */ -enum CriticalErrorCode { - NoError = 0, - - /// A software bug was detected while trying to send lora packets - ErrTxWatchdog = 1, - - /// A software bug was detected on entry to sleep - ErrSleepEnterWait = 2, - - /// No Lora radio hardware could be found - ErrNoRadio = 3, - - /// Not normally used - ErrUnspecified = 4, - - /// We failed while configuring a UBlox GPS - ErrUBloxInitFailed = 5, - - /// This board was expected to have a power management chip and it is missing or broken - ErrNoAXP192 = 6 - }; +#include "mesh/mesh.pb.h" // For CriticalErrorCode /// Record an error that should be reported via analytics -void recordCriticalError(CriticalErrorCode code, uint32_t address = 0); +void recordCriticalError(CriticalErrorCode code = CriticalErrorCode_Unspecified, uint32_t address = 0); diff --git a/src/gps/UBloxGPS.cpp b/src/gps/UBloxGPS.cpp index 7aa29b01..8fda6bc4 100644 --- a/src/gps/UBloxGPS.cpp +++ b/src/gps/UBloxGPS.cpp @@ -43,7 +43,7 @@ bool UBloxGPS::setupGPS() DEBUG_MSG("Connected to UBLOX GPS successfully\n"); if (!setUBXMode()) - recordCriticalError(ErrUBloxInitFailed); // Don't halt the boot if saving the config fails, but do report the bug + recordCriticalError(CriticalErrorCode_UBloxInitFailed); // Don't halt the boot if saving the config fails, but do report the bug return true; } else { diff --git a/src/main.cpp b/src/main.cpp index 7d27ad63..64851022 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -428,7 +428,7 @@ void setup() // Do this after service.init (because that clears error_code) #ifdef AXP192_SLAVE_ADDRESS if(!axp192_found) - recordCriticalError(ErrNoAXP192); // Record a hardware fault for missing hardware + recordCriticalError(CriticalErrorCode_NoAXP192); // Record a hardware fault for missing hardware #endif // Don't call screen setup until after nodedb is setup (because we need @@ -497,7 +497,7 @@ void setup() initWifi(forceSoftAP); if (!rIf) - recordCriticalError(ErrNoRadio); + recordCriticalError(CriticalErrorCode_NoRadio); else router->addInterface(rIf); diff --git a/src/mesh/NodeDB.cpp b/src/mesh/NodeDB.cpp index 9f4c144d..16ea5c16 100644 --- a/src/mesh/NodeDB.cpp +++ b/src/mesh/NodeDB.cpp @@ -271,7 +271,7 @@ void NodeDB::init() myNodeInfo.node_num_bits = sizeof(NodeNum) * 8; myNodeInfo.packet_id_bits = sizeof(PacketId) * 8; - myNodeInfo.error_code = NoError; // For the error code, only show values from this boot (discard value from flash) + myNodeInfo.error_code = CriticalErrorCode_None; // For the error code, only show values from this boot (discard value from flash) myNodeInfo.error_address = 0; // likewise - we always want the app requirements to come from the running appload diff --git a/src/mesh/mesh.pb.c b/src/mesh/mesh.pb.c index ca17e720..1fae4453 100644 --- a/src/mesh/mesh.pb.c +++ b/src/mesh/mesh.pb.c @@ -42,7 +42,7 @@ PB_BIND(MyNodeInfo, MyNodeInfo, AUTO) PB_BIND(DeviceState, DeviceState, 2) -PB_BIND(DebugString, DebugString, 2) +PB_BIND(LogRecord, LogRecord, AUTO) PB_BIND(FromRadio, FromRadio, 2) @@ -58,3 +58,5 @@ PB_BIND(ToRadio, ToRadio, 2) + + diff --git a/src/mesh/mesh.pb.h b/src/mesh/mesh.pb.h index 92bab3ae..fd382765 100644 --- a/src/mesh/mesh.pb.h +++ b/src/mesh/mesh.pb.h @@ -48,6 +48,16 @@ typedef enum _LocationSharing { LocationSharing_LocDisabled = 2 } LocationSharing; +typedef enum _CriticalErrorCode { + CriticalErrorCode_None = 0, + CriticalErrorCode_TxWatchdog = 1, + CriticalErrorCode_SleepEnterWait = 2, + CriticalErrorCode_NoRadio = 3, + CriticalErrorCode_Unspecified = 4, + CriticalErrorCode_UBloxInitFailed = 5, + CriticalErrorCode_NoAXP192 = 6 +} CriticalErrorCode; + typedef enum _ChannelSettings_ModemConfig { ChannelSettings_ModemConfig_Bw125Cr45Sf128 = 0, ChannelSettings_ModemConfig_Bw500Cr45Sf128 = 1, @@ -55,6 +65,16 @@ typedef enum _ChannelSettings_ModemConfig { ChannelSettings_ModemConfig_Bw125Cr48Sf4096 = 3 } ChannelSettings_ModemConfig; +typedef enum _LogRecord_Level { + LogRecord_Level_UNSET = 0, + LogRecord_Level_CRITICAL = 50, + LogRecord_Level_ERROR = 40, + LogRecord_Level_WARNING = 30, + LogRecord_Level_INFO = 20, + LogRecord_Level_DEBUG = 10, + LogRecord_Level_TRACE = 5 +} LogRecord_Level; + /* Struct definitions */ typedef PB_BYTES_ARRAY_T(32) ChannelSettings_psk_t; typedef struct _ChannelSettings { @@ -74,9 +94,12 @@ typedef struct _Data { Data_payload_t payload; } Data; -typedef struct _DebugString { - char message[256]; -} DebugString; +typedef struct _LogRecord { + char message[64]; + uint32_t time; + char source[8]; + LogRecord_Level level; +} LogRecord; typedef struct _MyNodeInfo { uint32_t my_node_num; @@ -85,7 +108,7 @@ typedef struct _MyNodeInfo { char region[12]; char hw_model[16]; char firmware_version[12]; - uint32_t error_code; + CriticalErrorCode error_code; uint32_t error_address; uint32_t error_count; uint32_t packet_id_bits; @@ -226,7 +249,7 @@ typedef struct _FromRadio { MyNodeInfo my_info; NodeInfo node_info; RadioConfig radio; - DebugString debug_string; + LogRecord log_record; uint32_t config_complete_id; bool rebooted; ChannelSettings secondary_channel; @@ -265,10 +288,18 @@ typedef struct _ToRadio { #define _LocationSharing_MAX LocationSharing_LocDisabled #define _LocationSharing_ARRAYSIZE ((LocationSharing)(LocationSharing_LocDisabled+1)) +#define _CriticalErrorCode_MIN CriticalErrorCode_None +#define _CriticalErrorCode_MAX CriticalErrorCode_NoAXP192 +#define _CriticalErrorCode_ARRAYSIZE ((CriticalErrorCode)(CriticalErrorCode_NoAXP192+1)) + #define _ChannelSettings_ModemConfig_MIN ChannelSettings_ModemConfig_Bw125Cr45Sf128 #define _ChannelSettings_ModemConfig_MAX ChannelSettings_ModemConfig_Bw125Cr48Sf4096 #define _ChannelSettings_ModemConfig_ARRAYSIZE ((ChannelSettings_ModemConfig)(ChannelSettings_ModemConfig_Bw125Cr48Sf4096+1)) +#define _LogRecord_Level_MIN LogRecord_Level_UNSET +#define _LogRecord_Level_MAX LogRecord_Level_CRITICAL +#define _LogRecord_Level_ARRAYSIZE ((LogRecord_Level)(LogRecord_Level_CRITICAL+1)) + #ifdef __cplusplus extern "C" { @@ -285,9 +316,9 @@ extern "C" { #define RadioConfig_init_default {false, RadioConfig_UserPreferences_init_default, false, ChannelSettings_init_default} #define RadioConfig_UserPreferences_init_default {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, "", "", 0, _RegionCode_MIN, _LocationSharing_MIN, _GpsOperation_MIN, 0, 0, 0, 0, 0, 0, 0, 0, {0, 0, 0}} #define NodeInfo_init_default {0, false, User_init_default, false, Position_init_default, 0, 0} -#define MyNodeInfo_init_default {0, 0, 0, "", "", "", 0, 0, 0, 0, 0, 0, 0, 0} +#define MyNodeInfo_init_default {0, 0, 0, "", "", "", _CriticalErrorCode_MIN, 0, 0, 0, 0, 0, 0, 0} #define DeviceState_init_default {false, RadioConfig_init_default, false, MyNodeInfo_init_default, false, User_init_default, 0, {NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default, NodeInfo_init_default}, 0, {MeshPacket_init_default}, false, MeshPacket_init_default, 0, 0, 0, 0, {ChannelSettings_init_default, ChannelSettings_init_default, ChannelSettings_init_default, ChannelSettings_init_default}} -#define DebugString_init_default {""} +#define LogRecord_init_default {"", 0, "", _LogRecord_Level_MIN} #define FromRadio_init_default {0, 0, {MeshPacket_init_default}} #define ToRadio_init_default {0, {MeshPacket_init_default}} #define Position_init_zero {0, 0, 0, 0, 0} @@ -300,9 +331,9 @@ extern "C" { #define RadioConfig_init_zero {false, RadioConfig_UserPreferences_init_zero, false, ChannelSettings_init_zero} #define RadioConfig_UserPreferences_init_zero {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, "", "", 0, _RegionCode_MIN, _LocationSharing_MIN, _GpsOperation_MIN, 0, 0, 0, 0, 0, 0, 0, 0, {0, 0, 0}} #define NodeInfo_init_zero {0, false, User_init_zero, false, Position_init_zero, 0, 0} -#define MyNodeInfo_init_zero {0, 0, 0, "", "", "", 0, 0, 0, 0, 0, 0, 0, 0} +#define MyNodeInfo_init_zero {0, 0, 0, "", "", "", _CriticalErrorCode_MIN, 0, 0, 0, 0, 0, 0, 0} #define DeviceState_init_zero {false, RadioConfig_init_zero, false, MyNodeInfo_init_zero, false, User_init_zero, 0, {NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero, NodeInfo_init_zero}, 0, {MeshPacket_init_zero}, false, MeshPacket_init_zero, 0, 0, 0, 0, {ChannelSettings_init_zero, ChannelSettings_init_zero, ChannelSettings_init_zero, ChannelSettings_init_zero}} -#define DebugString_init_zero {""} +#define LogRecord_init_zero {"", 0, "", _LogRecord_Level_MIN} #define FromRadio_init_zero {0, 0, {MeshPacket_init_zero}} #define ToRadio_init_zero {0, {MeshPacket_init_zero}} @@ -317,7 +348,10 @@ extern "C" { #define ChannelSettings_channel_num_tag 9 #define Data_portnum_tag 1 #define Data_payload_tag 2 -#define DebugString_message_tag 1 +#define LogRecord_message_tag 1 +#define LogRecord_time_tag 2 +#define LogRecord_source_tag 3 +#define LogRecord_level_tag 4 #define MyNodeInfo_my_node_num_tag 1 #define MyNodeInfo_has_gps_tag 2 #define MyNodeInfo_num_channels_tag 3 @@ -410,7 +444,7 @@ extern "C" { #define FromRadio_my_info_tag 3 #define FromRadio_node_info_tag 4 #define FromRadio_radio_tag 6 -#define FromRadio_debug_string_tag 7 +#define FromRadio_log_record_tag 7 #define FromRadio_config_complete_id_tag 8 #define FromRadio_rebooted_tag 9 #define FromRadio_secondary_channel_tag 10 @@ -550,7 +584,7 @@ X(a, STATIC, SINGULAR, INT32, num_channels, 3) \ X(a, STATIC, SINGULAR, STRING, region, 4) \ X(a, STATIC, SINGULAR, STRING, hw_model, 5) \ X(a, STATIC, SINGULAR, STRING, firmware_version, 6) \ -X(a, STATIC, SINGULAR, UINT32, error_code, 7) \ +X(a, STATIC, SINGULAR, UENUM, error_code, 7) \ X(a, STATIC, SINGULAR, UINT32, error_address, 8) \ X(a, STATIC, SINGULAR, UINT32, error_count, 9) \ X(a, STATIC, SINGULAR, UINT32, packet_id_bits, 10) \ @@ -582,10 +616,13 @@ X(a, STATIC, REPEATED, MESSAGE, secondary_channels, 12) #define DeviceState_rx_text_message_MSGTYPE MeshPacket #define DeviceState_secondary_channels_MSGTYPE ChannelSettings -#define DebugString_FIELDLIST(X, a) \ -X(a, STATIC, SINGULAR, STRING, message, 1) -#define DebugString_CALLBACK NULL -#define DebugString_DEFAULT NULL +#define LogRecord_FIELDLIST(X, a) \ +X(a, STATIC, SINGULAR, STRING, message, 1) \ +X(a, STATIC, SINGULAR, FIXED32, time, 2) \ +X(a, STATIC, SINGULAR, STRING, source, 3) \ +X(a, STATIC, SINGULAR, UENUM, level, 4) +#define LogRecord_CALLBACK NULL +#define LogRecord_DEFAULT NULL #define FromRadio_FIELDLIST(X, a) \ X(a, STATIC, SINGULAR, UINT32, num, 1) \ @@ -593,7 +630,7 @@ X(a, STATIC, ONEOF, MESSAGE, (variant,packet,variant.packet), 2) \ X(a, STATIC, ONEOF, MESSAGE, (variant,my_info,variant.my_info), 3) \ X(a, STATIC, ONEOF, MESSAGE, (variant,node_info,variant.node_info), 4) \ X(a, STATIC, ONEOF, MESSAGE, (variant,radio,variant.radio), 6) \ -X(a, STATIC, ONEOF, MESSAGE, (variant,debug_string,variant.debug_string), 7) \ +X(a, STATIC, ONEOF, MESSAGE, (variant,log_record,variant.log_record), 7) \ X(a, STATIC, ONEOF, UINT32, (variant,config_complete_id,variant.config_complete_id), 8) \ X(a, STATIC, ONEOF, BOOL, (variant,rebooted,variant.rebooted), 9) \ X(a, STATIC, ONEOF, MESSAGE, (variant,secondary_channel,variant.secondary_channel), 10) @@ -603,7 +640,7 @@ X(a, STATIC, ONEOF, MESSAGE, (variant,secondary_channel,variant.secondary_ #define FromRadio_variant_my_info_MSGTYPE MyNodeInfo #define FromRadio_variant_node_info_MSGTYPE NodeInfo #define FromRadio_variant_radio_MSGTYPE RadioConfig -#define FromRadio_variant_debug_string_MSGTYPE DebugString +#define FromRadio_variant_log_record_MSGTYPE LogRecord #define FromRadio_variant_secondary_channel_MSGTYPE ChannelSettings #define ToRadio_FIELDLIST(X, a) \ @@ -629,7 +666,7 @@ extern const pb_msgdesc_t RadioConfig_UserPreferences_msg; extern const pb_msgdesc_t NodeInfo_msg; extern const pb_msgdesc_t MyNodeInfo_msg; extern const pb_msgdesc_t DeviceState_msg; -extern const pb_msgdesc_t DebugString_msg; +extern const pb_msgdesc_t LogRecord_msg; extern const pb_msgdesc_t FromRadio_msg; extern const pb_msgdesc_t ToRadio_msg; @@ -646,7 +683,7 @@ extern const pb_msgdesc_t ToRadio_msg; #define NodeInfo_fields &NodeInfo_msg #define MyNodeInfo_fields &MyNodeInfo_msg #define DeviceState_fields &DeviceState_msg -#define DebugString_fields &DebugString_msg +#define LogRecord_fields &LogRecord_msg #define FromRadio_fields &FromRadio_msg #define ToRadio_fields &ToRadio_msg @@ -661,9 +698,9 @@ extern const pb_msgdesc_t ToRadio_msg; #define RadioConfig_size 308 #define RadioConfig_UserPreferences_size 219 #define NodeInfo_size 132 -#define MyNodeInfo_size 110 -#define DeviceState_size 5818 -#define DebugString_size 258 +#define MyNodeInfo_size 106 +#define DeviceState_size 5814 +#define LogRecord_size 81 #define FromRadio_size 329 #define ToRadio_size 323 diff --git a/src/mesh/portnums.pb.h b/src/mesh/portnums.pb.h index 703daa91..bef19c12 100644 --- a/src/mesh/portnums.pb.h +++ b/src/mesh/portnums.pb.h @@ -17,14 +17,14 @@ typedef enum _PortNum { PortNum_POSITION_APP = 3, PortNum_NODEINFO_APP = 4, PortNum_REPLY_APP = 32, - PortNum_PRIVATE_APP = 256, - PortNum_IP_TUNNEL_APP = 1024 + PortNum_IP_TUNNEL_APP = 33, + PortNum_PRIVATE_APP = 256 } PortNum; /* Helper constants for enums */ #define _PortNum_MIN PortNum_UNKNOWN_APP -#define _PortNum_MAX PortNum_IP_TUNNEL_APP -#define _PortNum_ARRAYSIZE ((PortNum)(PortNum_IP_TUNNEL_APP+1)) +#define _PortNum_MAX PortNum_PRIVATE_APP +#define _PortNum_ARRAYSIZE ((PortNum)(PortNum_PRIVATE_APP+1)) #ifdef __cplusplus diff --git a/src/sleep.cpp b/src/sleep.cpp index 60bf911f..ae175eed 100644 --- a/src/sleep.cpp +++ b/src/sleep.cpp @@ -127,7 +127,7 @@ static void waitEnterSleep() delay(100); // Kinda yucky - wait until radio says say we can shutdown (finished in process sends/receives) if (millis() - now > 30 * 1000) { // If we wait too long just report an error and go to sleep - recordCriticalError(ErrSleepEnterWait); + recordCriticalError(CriticalErrorCode_SleepEnterWait); assert(0); // FIXME - for now we just restart, need to fix bug #167 break; } From 2edc6b363d39e94789dd42358361d9c22d05b727 Mon Sep 17 00:00:00 2001 From: Kevin Hester Date: Sun, 27 Dec 2020 12:53:23 +0800 Subject: [PATCH 08/10] fix #587 thanks @cronyx --- bin/platformio-custom.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/platformio-custom.py b/bin/platformio-custom.py index ec9dd454..265158f7 100644 --- a/bin/platformio-custom.py +++ b/bin/platformio-custom.py @@ -12,5 +12,5 @@ print("Using meshtastic platform-custom.py, firmare version " + verStr) # General options that are passed to the C and C++ compilers projenv.Append(CCFLAGS=[ - f"-DAPP_VERSION={verStr}" + "-DAPP_VERSION=" + verStr ]) From 21570fc24fbdecf6ace4d40fdfd3e7929bbe47f0 Mon Sep 17 00:00:00 2001 From: Kevin Hester Date: Sun, 27 Dec 2020 12:54:44 +0800 Subject: [PATCH 09/10] fix #540 use gps to fix rtc clock drift every 12 hrs --- src/gps/RTC.cpp | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/src/gps/RTC.cpp b/src/gps/RTC.cpp index bc65dea1..387f4c0c 100644 --- a/src/gps/RTC.cpp +++ b/src/gps/RTC.cpp @@ -31,9 +31,23 @@ void readFromRTC() /// If we haven't yet set our RTC this boot, set it from a GPS derived time bool perhapsSetRTC(RTCQuality q, const struct timeval *tv) { + static uint32_t lastSetMsec = 0; + uint32_t now = millis(); + + bool shouldSet; if (q > currentQuality) { - currentQuality = q; - DEBUG_MSG("Setting RTC %ld secs\n", tv->tv_sec); + shouldSet = true; + DEBUG_MSG("Upgrading time to RTC %ld secs (quality %d)\n", tv->tv_sec, q); + } else if(q == RTCQualityGPS && (now - lastSetMsec) > (12 * 60 * 60 * 1000L)) { + // Every 12 hrs we will slam in a new GPS time, to correct for local RTC clock drift + shouldSet = true; + DEBUG_MSG("Reapplying GPS time to correct clock drift %ld secs\n", tv->tv_sec); + } + else + shouldSet = false; + + if (shouldSet) { + lastSetMsec = now; #ifndef NO_ESP32 settimeofday(tv, NULL); #else From 186a52172c35b15e8dc180acea4c92adca912cb9 Mon Sep 17 00:00:00 2001 From: Kevin Hester Date: Sun, 27 Dec 2020 13:09:20 +0800 Subject: [PATCH 10/10] fix #577 don't make invalid radio settings reboot the board instead raise a critical fault (note though: this is still not ideal because the radio will be in an undefined state until valid settings are used) --- proto | 2 +- src/mesh/RF95Interface.cpp | 13 ++++++++----- src/mesh/SX1262Interface.cpp | 9 +++++---- src/mesh/mesh.pb.h | 7 ++++--- 4 files changed, 18 insertions(+), 13 deletions(-) diff --git a/proto b/proto index 9a7d8a03..323b814f 160000 --- a/proto +++ b/proto @@ -1 +1 @@ -Subproject commit 9a7d8a03cb051eb42787d4a06836b109677d8ef1 +Subproject commit 323b814f4392ae0f9c42a0f14557c6b9333efce3 diff --git a/src/mesh/RF95Interface.cpp b/src/mesh/RF95Interface.cpp index 9410ad65..60596468 100644 --- a/src/mesh/RF95Interface.cpp +++ b/src/mesh/RF95Interface.cpp @@ -1,6 +1,7 @@ #include "RF95Interface.h" #include "MeshRadio.h" // kinda yucky, but we need to know which region we are in #include "RadioLibRF95.h" +#include "error.h" #include #define MAX_POWER 20 @@ -85,6 +86,8 @@ void INTERRUPT_ATTR RF95Interface::disableInterrupt() lora->clearDio0Action(); } + + bool RF95Interface::reconfigure() { applyModemConfig(); @@ -94,13 +97,13 @@ bool RF95Interface::reconfigure() // configure publicly accessible settings int err = lora->setSpreadingFactor(sf); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); err = lora->setBandwidth(bw); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); err = lora->setCodingRate(cr); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); err = lora->setSyncWord(syncWord); assert(err == ERR_NONE); @@ -112,12 +115,12 @@ bool RF95Interface::reconfigure() assert(err == ERR_NONE); err = lora->setFrequency(freq); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); if (power > MAX_POWER) // This chip has lower power limits than some power = MAX_POWER; err = lora->setOutputPower(power); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); startReceive(); // restart receiving diff --git a/src/mesh/SX1262Interface.cpp b/src/mesh/SX1262Interface.cpp index 0a23261b..0bff4736 100644 --- a/src/mesh/SX1262Interface.cpp +++ b/src/mesh/SX1262Interface.cpp @@ -1,4 +1,5 @@ #include "SX1262Interface.h" +#include "error.h" #include // Particular boards might define a different max power based on what their hardware can do @@ -78,13 +79,13 @@ bool SX1262Interface::reconfigure() // configure publicly accessible settings int err = lora.setSpreadingFactor(sf); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); err = lora.setBandwidth(bw); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); err = lora.setCodingRate(cr); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); // Hmm - seems to lower SNR when the signal levels are high. Leaving off for now... err = lora.setRxGain(true); @@ -100,7 +101,7 @@ bool SX1262Interface::reconfigure() assert(err == ERR_NONE); err = lora.setFrequency(freq); - assert(err == ERR_NONE); + if(err != ERR_NONE) recordCriticalError(CriticalErrorCode_InvalidRadioSetting); if (power > 22) // This chip has lower power limits than some power = 22; diff --git a/src/mesh/mesh.pb.h b/src/mesh/mesh.pb.h index fd382765..1da4dd3c 100644 --- a/src/mesh/mesh.pb.h +++ b/src/mesh/mesh.pb.h @@ -55,7 +55,8 @@ typedef enum _CriticalErrorCode { CriticalErrorCode_NoRadio = 3, CriticalErrorCode_Unspecified = 4, CriticalErrorCode_UBloxInitFailed = 5, - CriticalErrorCode_NoAXP192 = 6 + CriticalErrorCode_NoAXP192 = 6, + CriticalErrorCode_InvalidRadioSetting = 7 } CriticalErrorCode; typedef enum _ChannelSettings_ModemConfig { @@ -289,8 +290,8 @@ typedef struct _ToRadio { #define _LocationSharing_ARRAYSIZE ((LocationSharing)(LocationSharing_LocDisabled+1)) #define _CriticalErrorCode_MIN CriticalErrorCode_None -#define _CriticalErrorCode_MAX CriticalErrorCode_NoAXP192 -#define _CriticalErrorCode_ARRAYSIZE ((CriticalErrorCode)(CriticalErrorCode_NoAXP192+1)) +#define _CriticalErrorCode_MAX CriticalErrorCode_InvalidRadioSetting +#define _CriticalErrorCode_ARRAYSIZE ((CriticalErrorCode)(CriticalErrorCode_InvalidRadioSetting+1)) #define _ChannelSettings_ModemConfig_MIN ChannelSettings_ModemConfig_Bw125Cr45Sf128 #define _ChannelSettings_ModemConfig_MAX ChannelSettings_ModemConfig_Bw125Cr48Sf4096