mqtt: automatically start or stop as neede & attempt reconnect

docs/software/TODO.md

@@ -6,6 +6,7 @@ You probably don't care about this section - skip to the next one.
 
 * android speed settings https://github.com/meshtastic/Meshtastic-Android/issues/271
 * fix heltec battery scaling
+
 * DONE remote admin busted? 
 * DONE check android code - @havealoha comments about odd sleep behavior
 * ABANDONED test github actions locally on linux
@@ -51,33 +52,6 @@ You probably don't care about this section - skip to the next one.
 * DONE show GPS time only if we know what global time is
 * DONE android should always provide time to nodes - so that it is easier for the mesh to learn the current time
 
-## MQTT
-
-* reply to MC
-* reply to question about MQTT
-* add test case of preencrypting in the python tool (and sending through a node that lacks keys)
-* leave encrypted messages as forwarded (need fixes on both tx and rx sides)
-* DONE have sim provide a fake wifi connection status saying connected
-* DONE don't start MQTT if we don't have wifi connected
-* have plugin send uplinks from mesh
-* have plugin send downlinks to mesh
-* DONE don't decrypt messages before uplinking them to MQTT (move out of plugin)
-* mqtt.meshtastic.org should have VERY basic auth at launch (to prevent abuse)
-* make a GlobalChat channel as an initial test (with a well known AES128 key), figure out how globally unique IDs work
-* Give place in android app for users to select which channel they are sending on (and which channels they are watching)
-* attempt reconnect to server if internet connectivity changes
-* don't bother contacting server if we don't have any uplink/downlink channels
-* DONE test on ESP32
-* no need for python gateway to web initially: because both the simulator and ESP32 can talk wifi directly
-* if simmesh_name is set in preferences, create the MQTTSimInterface using that as the global channel_id
-* figure out how to use MQTT for simulator mesh network, use a special simmesh_name global channel_id? (because this would allow all nodes in simnet_xxx to subscribe only to those packets)
-* figure out legality of hosting public mqtt servers with chat msgs
-* DONE do initial development inside of portduino
-* DONE do as much possible on the device side (so we can eventually just have ESP32 talk directly to server)
-* DONE add mqtt_server to radio prefs
-* eventually add a MQTTPacket on the ToRadio & FromRadio links
-* LATER: an android gateway would be useful
-
 ## Multichannel support
 
 * DONE cleanup the external notification and serial plugins

docs/software/crypto.md

@@ -36,6 +36,14 @@ If you are reviewing our implementation, this is a brief statement of our method
 - Each 16 byte BLOCK for a packet has an incrementing COUNTER. COUNTER starts at zero for the first block of each packet.
 - The IV for each block is constructed by concatenating the NONCE as the upper 96 bits of the IV and the COUNTER as the bottom 32 bits. Since our packets are small counter portion will really never be higher than 32 (five bits).
 
+### Details on the nonce encoding
+
+(For those porting to other architectures)
+Bytes 0-3 of the nonce are the "packet number" in little-endian order
+Bytes 4-7 are currently zero
+Bytes 8-11 are the sending node ID in little-endian order
+Bytes 12-15 are currently zero
+
 ## Comments from reviewer #1
 
 This reviewer is a cryptography professional, but would like to remain anonymous. We thank them for their comments ;-):

docs/software/mqtt.md

@@ -17,6 +17,7 @@
     - [MQTTSimInterface](#mqttsiminterface)
   - [Web services](#web-services)
     - [Public MQTT broker service](#public-mqtt-broker-service)
+      - [Broker selection](#broker-selection)
     - [Admin service](#admin-service)
     - [Riot.im messaging bridge](#riotim-messaging-bridge)
   - [Deprecated concepts](#deprecated-concepts)
@@ -139,6 +140,12 @@ An existing public [MQTT broker](https://mosquitto.org/) will be the default for
 
 FIXME - figure out how to avoid impersonation (because we are initially using a public mqtt server with no special security options).  FIXME, include some ideas on this in the ServiceEnvelope documentation.
 
+#### Broker selection
+
+On a previous project I used mosqitto, which I liked, but the admin interface for programmatically managing access was ugly.  [This](https://www.openlogic.com/blog/activemq-vs-rabbitmq) article makes me think RabbitMQ might be best for us.
+
+Initially I will try to avoid using any non MQTT broker/library/API
+
 ### Admin service
 
 (This is a WIP draft collection of not complete ideas)
@@ -155,7 +162,7 @@ Topics:
 
 Operations provided via the ToAdmin/ToNode protocol:
 
-* Request global channel ID (request a new channel ID)
+* Register a global channel ID (request a new channel ID).  Optionally include the AES key if you would like the web service to automatically decrypt in the cloud
 * Request gateway ID - the response is used to re-sign in to the broker.  
 
 Possibly might need public key encryption for the gateway request?  Since the response is sent to the mesh/to endpoint?  I would really like to use MQTT for all comms so I don't need yet another protocol/port from the device.

src/mqtt/MQTT.cpp

@@ -20,14 +20,7 @@ void mqttCallback(char *topic, byte *payload, unsigned int length)
 
 void mqttInit()
 {
-    // FIXME, for now we require the user to specifically set a MQTT server (till tested)
-    if (radioConfig.preferences.mqtt_disabled || !*radioConfig.preferences.mqtt_server)
-        DEBUG_MSG("MQTT disabled...\n");
-    else if (!WiFi.isConnected())
-        DEBUG_MSG("WiFi is not connected, can not start MQTT\n");
-    else {
-        new MQTT();
-    }
+    new MQTT();
 }
 
 MQTT::MQTT() : concurrency::OSThread("mqtt"), pubSub(mqttClient)
@@ -35,6 +28,11 @@ MQTT::MQTT() : concurrency::OSThread("mqtt"), pubSub(mqttClient)
     assert(!mqtt);
     mqtt = this;
 
+    pubSub.setCallback(mqttCallback);
+}
+
+void MQTT::reconnect()
+{
     // pubSub.setServer("devsrv.ezdevice.net", 1883); or 192.168.10.188
     const char *serverAddr = "test.mosquitto.org"; // "mqtt.meshtastic.org"; // default hostname
 
@@ -42,9 +40,8 @@ MQTT::MQTT() : concurrency::OSThread("mqtt"), pubSub(mqttClient)
         serverAddr = radioConfig.preferences.mqtt_server; // Override the default
 
     pubSub.setServer(serverAddr, 1883);
-    pubSub.setCallback(mqttCallback);
 
-    DEBUG_MSG("Connecting to MQTT server: %s\n", serverAddr);
+    DEBUG_MSG("Connecting to MQTT server\n", serverAddr);
     auto myStatus = (statusTopic + owner.id);
     // bool connected = pubSub.connect(nodeId.c_str(), "meshdev", "apes4cats", myStatus.c_str(), 1, true, "offline");
     bool connected = pubSub.connect(owner.id, myStatus.c_str(), 1, true, "offline");
@@ -61,16 +58,53 @@ MQTT::MQTT() : concurrency::OSThread("mqtt"), pubSub(mqttClient)
         /// FIXME, include more information in the status text
         bool ok = pubSub.publish(myStatus.c_str(), "online", true);
         DEBUG_MSG("published %d\n", ok);
+    } else
+        DEBUG_MSG("Failed to contact MQTT server...\n");
+}
+
+bool MQTT::wantsLink() const
+{
+    bool hasChannel = false;
+
+    if (radioConfig.preferences.mqtt_disabled) {
+        // DEBUG_MSG("MQTT disabled...\n");
+    } else {
+        // No need for link if no channel needed it
+        size_t numChan = channels.getNumChannels();
+        for (size_t i = 0; i < numChan; i++) {
+            auto &ch = channels.getByIndex(i);
+            if (ch.settings.uplink_enabled || ch.settings.downlink_enabled) {
+                hasChannel = true;
+                break;
+            }
+        }
     }
+
+    return hasChannel && WiFi.isConnected();
 }
 
 int32_t MQTT::runOnce()
 {
-    // If connected poll rapidly, otherwise sleep forever
-    if (!pubSub.loop())
-        enabled = false;
+    bool wantConnection = wantsLink();
 
-    return 20;
+    // If connected poll rapidly, otherwise only occasionally check for a wifi connection change and ability to contact server
+    if (!pubSub.loop()) {
+        if (wantConnection) {
+            reconnect();
+
+            // If we succeeded, start reading rapidly, else try again in 30 seconds (TCP connections are EXPENSIVE so try rarely)
+            return pubSub.connected() ? 20 : 30000;
+        } else
+            return 5000; // If we don't want connection now, check again in 5 secs
+    } else {
+        // we are connected to server, check often for new requests on the TCP port
+        if (!wantConnection) {
+            DEBUG_MSG("MQTT link not needed, dropping\n");
+            pubSub.disconnect();
+        }
+
+        return 20;
+    }
 }
 
 void MQTT::onSend(const MeshPacket &mp, ChannelIndex chIndex)

src/mqtt/MQTT.h

@@ -37,6 +37,14 @@ class MQTT : private concurrency::OSThread
 
   private:
     const char *getCryptTopic(const char *channelId);
+
+    /** return true if we have a channel that wants uplink/downlink
+     */
+    bool wantsLink() const;
+
+    /** Attempt to connect to server if necessary
+     */
+    void reconnect();
 };
 
 void mqttInit();