Merge remote-tracking branch 'origin/master' into fb_add_itron_ultramaxx

2024-01-18 20:12:59 +01:00 · 2024-01-18 20:12:59 +01:00 · f9c3fa4d64
commit f9c3fa4d64
--- a/.github/workflows/close_stale_issues.yml
+++ b/.github/workflows/close_stale_issues.yml
@ -10,7 +10,7 @@ jobs:
  stale:
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/stale@v8
+      - uses: actions/stale@v9
        with:
          stale-issue-message: 'This issue is stale because it has been open for 2 month with no activity. Remove stale label or comment or this will be closed in 1 month.'
          close-issue-message: 'This issue was closed because it has been stalled for 1 month with no activity.'
--- a/20
+++ b/20
@ -1,4 +1,24 @@

+ATTENTION! Wmbusmeters now use new -f option when starting rtl_wmbus. There is a
+warning if rtl_wmbus does not support the -f option and an upgrade is recommended.
+This option will cause rtl_wmbus to exit with an error if the rtl_sdr dongle stops sending data.
+This in turn will cause wmbusmeters to restart the pipeline.
+
+Up till now, the stderr from rtl_sdr has been sent to /dev/null. This is a problem
+since we cannot see any errors from rtl_sdr that could have caused it to stall.
+
+However the reason for /dev/null was this bug in rtl_sdr.
+https://github.com/osmocom/rtl-sdr/commit/142325a93c6ad70f851f43434acfdf75e12dfe03
+which prevented us from sending the rtl_sdr stderr to wmbusmeters.
+If we did, rtl_sdr went into a 100% cpu hang when we restarted a wmbusmeters daemon.
+
+A temporary workaround has been found that both sends the stderr output to wmbusmeters
+and permits the restart of the daemon. Stderr from rtl_sdr is now sent to
+/tmp/tmp.XXXXXXX_wmbusmeters_rtlsdr and then tailed into wmbusmeters.
+This is a temporary solution until the real rtl_sdr bugfix has propagated into enough distributions.
+
+Add second extension energy MWh VIF 7b00-7b01.
+
 Sunflowerenergias improved the iwmtx5 driver! Thanks Sunflowerenergias!

 Jacman777 improved the kamheat driver! Thanks Jacman777!
--- a/README.md
+++ b/README.md
@ -479,10 +479,24 @@ These telegrams are expected to have the data link layer crc bytes removed alrea

 `MAIN=/dev/ttyUSB0:mbus:2400`, assume ttyUSB0 is an serial to mbus-master converter.  The speed is set to 2400 bps.

-`rtlwmbus`, to spawn the background process: `rtl_sdr -f 868.625M -s 1600000 - 2>/dev/null | rtl_wmbus -s`
+`rtlwmbus`, to spawn the background process: `rtl_sdr -f 868.625M -s 1600000 - 2>/dev/null | rtl_wmbus -f -s`
 for each attached rtlsdr dongle. This will listen to S1,T1 and C1 meters in parallel.

-Note that this uses a noticeable amount of CPU time by rtl_wmbus.
+For the moment, it is necessary to send the stderr to a file (/dev/null) because of a bug:
+https://github.com/osmocom/rtl-sdr/commit/142325a93c6ad70f851f43434acfdf75e12dfe03
+
+Until this bug fix has propagated into Debian/Fedora etc, wmbusmeters uses a tmp file
+to see the stderr output from rtl_sdr. This tmp file is created in /tmp and will
+generate 420 bytes of data once ever 23 hours.
+
+The current command line used by wmbusmeters to start the rtl_wmbus pipeline is therefore a bit longer:
+```
+ERRFILE=$(mktemp --suffix=_wmbusmeters_rtlsdr) ;
+echo ERRFILE=$ERRFILE ;  date -Iseconds > $ERRFILE ;
+tail -f $ERRFILE & /usr/bin/rtl_sdr  -d 0 -f 868.625M -s 1.6e6 - 2>>$ERRFILE | /usr/bin/rtl_wmbus -s -f
+```
+
+Note that the standard -s option uses a noticeable amount of CPU time by rtl_wmbus.
 You can therefore use a tailored rtl_wmbus command that is more suitable for your needs.

 `rtlwmbus:CMD(<command line>)`, to specify the entire background
@ -492,9 +506,10 @@ The command line cannot contain parentheses.
 Likewise for rtl433.

 Here is an example command line that reduces the rtl_wmbus CPU usage if you only need T1/C1 telegrams.
-It disable S1 decoding (`-p s`) and trades lower cpu usage for reception performance (`-a`):
+It disable S1 decoding (`-p s`) and trades lower cpu usage for reception performance (`-a`).
+You should always add the `-f` option to enable detection if rtl_sdr has stalled:

-`rtlwmbus:CMD(rtl_sdr -f 868.95M -s 1600000 - 2>/dev/null | rtl_wmbus -p s -a)`
+`rtlwmbus:CMD(rtl_sdr -f 868.95M -s 1600000 - 2>/dev/null | rtl_wmbus -p s -a -f)`

 `rtlwmbus(ppm=17)`, to tune your rtlsdr dongle accordingly.
 Use this to tune your dongle and at the same time listen to S1,T1 and C1.
@ -797,12 +812,12 @@ wmbusmeters --format=json --meterfiles /dev/ttyUSB0:im871a:c1 MyTapWater multica
 # Using wmbusmeters in a pipe

 ```shell
-rtl_sdr -f 868.625M -s 1600000 - 2>/dev/null | rtl_wmbus -s | wmbusmeters --format=json stdin:rtlwmbus MyMeter auto 12345678 NOKEY | ...more processing...
+rtl_sdr -f 868.625M -s 1600000 - 2>/dev/null | rtl_wmbus -f -s | wmbusmeters --format=json stdin:rtlwmbus MyMeter auto 12345678 NOKEY | ...more processing...
 ```

 Or you can send rtl_wmbus formatted telegrams using nc over UDP to wmbusmeters.
 ```shell
-rtl_sdr -f 868.95M -s 1600000 - 2>/dev/null | rtl_wmbus -p s -a | nc -u localhost 4444
+rtl_sdr -f 868.95M -s 1600000 - 2>/dev/null | rtl_wmbus -f -p s -a | nc -u localhost 4444
 ```

 And receive the telegrams with nc spawned by wmbusmeters.
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@ -1,22 +1,18 @@
-FROM multiarch/alpine:${TARGETARCH}${TARGETVARIANT}-latest-stable AS build
+FROM asymworks/multiarch-alpine:${TARGETARCH}${TARGETVARIANT}-latest-stable AS build
 RUN apk add --no-cache alpine-sdk gcc linux-headers librtlsdr-dev libxml2-dev cmake libusb-dev bash
 RUN git clone https://github.com/wmbusmeters/wmbusmeters.git && \
-    git clone https://github.com/weetmuts/rtl-wmbus.git && \
-    git clone https://github.com/merbanan/rtl_433.git
+    git clone https://github.com/weetmuts/rtl-wmbus.git
 WORKDIR /wmbusmeters
 RUN make
 WORKDIR /rtl-wmbus
 RUN make release && chmod 755 build/rtl_wmbus
-WORKDIR /rtl_433
-RUN mkdir build && cd build && cmake ../ && make

-FROM multiarch/alpine:${TARGETARCH}${TARGETVARIANT}-latest-stable as scratch
+FROM asymworks/multiarch-alpine:${TARGETARCH}${TARGETVARIANT}-latest-stable as scratch
 ENV QEMU_EXECVE=1
-RUN apk add --no-cache mosquitto-clients libstdc++ curl libusb rtl-sdr libxml2 netcat-openbsd
+RUN apk add --no-cache mosquitto-clients libstdc++ curl libusb rtl-sdr libxml2 netcat-openbsd rtl_433
 WORKDIR /wmbusmeters
 COPY --from=build /wmbusmeters/build/wmbusmeters /wmbusmeters/wmbusmeters
 COPY --from=build /rtl-wmbus/build/rtl_wmbus /usr/bin/rtl_wmbus
-COPY --from=build /rtl_433/build/src/rtl_433 /usr/bin/rtl_433
 COPY --from=build /wmbusmeters/docker/docker-entrypoint.sh /wmbusmeters/docker-entrypoint.sh
 VOLUME /wmbusmeters_data/
-CMD ["sh", "/wmbusmeters/docker-entrypoint.sh"]
+CMD ["sh", "/wmbusmeters/docker-entrypoint.sh"]
--- a/scripts/prepare_logfiles.sh
+++ b/scripts/prepare_logfiles.sh
@ -44,6 +44,7 @@ then
    postrotate
        /bin/kill -HUP \`cat /run/wmbusmeters/wmbusmeters.pid 2> /dev/null\` 2> /dev/null || true
    endscript
+}
 EOF
    echo "logrotate: created $ROOT/etc/logrotate.d/wmbusmeters"
 else
--- a/src/driver_iem3000.cc
+++ b/src/driver_iem3000.cc
@ -159,7 +159,7 @@ namespace
            VifScaling::None,
            FieldMatcher::build()
            .set(DifVifKey("04FFA015")),
-            Unit::FACTOR
+            Unit::NUMBER
            );

        addNumericFieldWithExtractor(
@ -170,7 +170,7 @@ namespace
            VifScaling::None,
            FieldMatcher::build()
            .set(DifVifKey("04FFA115")),
-            Unit::FACTOR
+            Unit::NUMBER
            );

        addNumericFieldWithExtractor(
@ -181,7 +181,7 @@ namespace
            VifScaling::None,
            FieldMatcher::build()
            .set(DifVifKey("04FFA215")),
-            Unit::FACTOR
+            Unit::NUMBER
            );

        addNumericFieldWithExtractor(
@ -192,7 +192,7 @@ namespace
            VifScaling::None,
            FieldMatcher::build()
            .set(DifVifKey("04FFA315")),
-            Unit::FACTOR
+            Unit::NUMBER
            );

       addStringFieldWithExtractorAndLookup(
@ -1569,7 +1569,7 @@ namespace

        addNumericFieldWithExtractor(
            "vts",
-            "Number of VT:s.",
+            "Number of voltaget transformers VT:s.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
@ -1578,7 +1578,7 @@ namespace

        addNumericFieldWithExtractor(
            "vt_primary",
-            "Primary VT.",
+            "Primary voltage transformer VT.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
@ -1588,7 +1588,7 @@ namespace

        addNumericFieldWithExtractor(
            "vt_secondary",
-            "Secondary VT.",
+            "Secondary voltage transformer VT.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
@ -1598,7 +1598,7 @@ namespace

        addNumericFieldWithExtractor(
            "cts",
-            "Number of CT:s.",
+            "Number of current transformers CT:s.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
@ -1607,7 +1607,7 @@ namespace

        addNumericFieldWithExtractor(
            "ct_primary",
-            "Primary CT.",
+            "Primary current transformer CT.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
@ -1617,7 +1617,7 @@ namespace

        addNumericFieldWithExtractor(
            "ct_secondary",
-            "Secondary CT.",
+            "Secondary current transformer CT.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
@ -1627,7 +1627,7 @@ namespace

        addNumericFieldWithExtractor(
            "vt_connection_type",
-            "VT connection type.",
+            "Voltage transformer connection type.",
            DEFAULT_PRINT_PROPERTIES,
            Quantity::Dimensionless,
            VifScaling::None,
--- a/src/driver_kamheat.cc
+++ b/src/driver_kamheat.cc
@ -50,8 +50,8 @@ namespace
        di.addDetection(MANUFACTURER_KAM, 0x0c,  0x34); // 403
        di.addDetection(MANUFACTURER_KAM, 0x0d,  0x34); // 403
        di.addDetection(MANUFACTURER_KAM, 0x04,  0x1c); // 602
-        di.addDetection(MANUFACTURER_KAM, 0x04, 0x35); // 603
-        di.addDetection(MANUFACTURER_KAM, 0x0c, 0x35); // 603
+        di.addDetection(MANUFACTURER_KAM, 0x04,  0x35); // 603
+        di.addDetection(MANUFACTURER_KAM, 0x0c,  0x35); // 603
        di.addDetection(MANUFACTURER_KAM, 0x04,  0x39); // 803

        di.setConstructor([](MeterInfo& mi, DriverInfo& di){ return shared_ptr<Meter>(new Driver(mi, di)); });
@ -59,7 +59,8 @@ namespace

    Driver::Driver(MeterInfo &mi, DriverInfo &di) : MeterCommonImplementation(mi, di)
    {
-        addOptionalCommonFields("on_time_h");
+        addOptionalCommonFields("fabrication_no,meter_datetime,on_time_h,on_time_at_error_h");
+        addOptionalFlowRelatedFields("flow_return_temperature_difference_c");

        // Technical Description Multical 603 page 116 section 7.7.2 Information code types on serial communication.
        addStringFieldWithExtractorAndLookup(
@ -348,3 +349,8 @@ namespace
 // telegram=|40442D2C0650216219048D2083A4E1162306FF78_040F2C3F000004FF07DBA40D0004FF08860B0D000414BA33140002FD170000043B620000000259A21E025DFA1B|
 // {"media":"heat","meter":"kamheat","name":"Kamstrup_402_wmbus","id":"62215006","forward_energy_m3c":894171,"return_energy_m3c":854918,"t1_temperature_c":78.42,"t2_temperature_c":71.62,"total_energy_consumption_kwh":44922.222222,"total_volume_m3":13239.62,"volume_flow_m3h":0.098,"status":"OK","timestamp":"1111-11-11T11:11:11Z"}
 // |Kamstrup_402_wmbus;62215006;44922.222222;13239.62;OK;1111-11-11 11:11.11
+
+// Test: Kamstrup_MC603_mbus kamheat 32323232 NOKEY
+// telegram=|68c9c96808e672323232322d2c35041900000004fB006083000004ff074006010004ff08299400000416984e010084401400000000848040140000000004225043000034221c0000000259c91f025d4f1102617a0e042e30020000142e65030000043c24050000143ce308000004ff2200000000046d2e2B0f3144fB00007d000044ff07Bdf9000044ff08308d00004416B73f0100c4401400000000c480401400000000542ed9020000543ce8090000426c013102ff1a011B0c783032858404ff16e5841e0004ff17c1d5B400a516|
+// {"fabrication_no": "84853230",  "flow_return_temperature_difference_c": 37.06,  "forward_energy_m3c": 67136,  "id": "32323232",  "max_flow_m3h": 22.75,  "max_power_kw": 869,  "media": "heat",  "meter": "kamheat",  "meter_datetime": "2024-01-15 11:46",  "name": "Kamstrup_MC603_mbus",  "on_time_at_error_h": 28,  "on_time_h": 17232,  "power_kw": 560,  "return_energy_m3c": 37929,  "status": "OK",  "t1_temperature_c": 81.37,  "t2_temperature_c": 44.31,  "target_date": "2024-01-01",  "target_energy_kwh": 3200000,  "target_volume_m3": 81847,  "timestamp": "1111-11-11T11:11:11Z",  "total_energy_consumption_kwh": 3363200,  "total_volume_m3": 85656,  "volume_flow_m3h": 13.16}
+// |Kamstrup_MC603_mbus;32323232;3363200;85656;OK;1111-11-11 11:11.11
--- a/src/dvparser.cc
+++ b/src/dvparser.cc
@ -123,7 +123,8 @@ bool isInsideVIFRange(Vif vif, VIFRange vif_range)
    {
        return
            isInsideVIFRange(vif, VIFRange::EnergyWh) ||
-            isInsideVIFRange(vif, VIFRange::EnergyMJ);
+            isInsideVIFRange(vif, VIFRange::EnergyMJ) ||
+            isInsideVIFRange(vif, VIFRange::EnergyMWh);
    }
    if (vif_range == VIFRange::AnyPowerVIF)
    {
--- a/src/dvparser.h
+++ b/src/dvparser.h
@ -47,6 +47,7 @@
    X(ActualityDuration,0x74,0x77, Quantity::Time, Unit::Hour) \
    X(FabricationNo,0x78,0x78, Quantity::Text, Unit::TXT) \
    X(EnhancedIdentification,0x79,0x79, Quantity::Text, Unit::TXT) \
+    X(EnergyMWh,0x7B00,0x7B01, Quantity::Energy, Unit::KWH) \
    X(RelativeHumidity,0x7B1A,0x7B1B, Quantity::RH, Unit::RH) \
    X(AccessNumber,0x7D08,0x7D08, Quantity::Counter, Unit::COUNTER) \
    X(Medium,0x7D09,0x7D09, Quantity::Text, Unit::TXT) \
--- a/src/wmbus.cc
+++ b/src/wmbus.cc
@ -2652,6 +2652,9 @@ string vifType(int vif)
    case 0x7E: return "Any VIF";
    case 0x7F: return "Manufacturer specific";

+    case 0x7B00: return "Active Energy 0.1 MWh";
+    case 0x7B01: return "Active Energy 1 MWh";
+
    case 0x7B1A: return "Relative humidity 0.1%";
    case 0x7B1B: return "Relative humidity 1%";

@ -2826,6 +2829,12 @@ double vifScale(int vif)
    case 0x76: return 1.0; // Actuality duration hours
    case 0x77: return (1.0/24.0); // Actuality duration days

+        // Active energy 0.1 or 1 MWh normalize to 100 KWh or 1000 KWh
+        // 7b00 33632 -> 3363.2 MWh -> 3363200 KWh
+        // 7b01 33632 -> 33632 MWh -> 33632000 KWh
+    case 0x7b00:
+    case 0x7b01: { double exp = (vif & 0x1)+2; return pow(10.0, -exp); }
+
        // relative humidity is a dimensionless value.
    case 0x7b1a: return 10.0; // Relative humidity 0.1 %
    case 0x7b1b: return 1.0;  // Relative humidity 1 %
--- a/src/wmbus_rtlwmbus.cc
+++ b/src/wmbus_rtlwmbus.cc
@ -20,8 +20,10 @@
 #include"wmbus_utils.h"
 #include"rtlsdr.h"
 #include"serial.h"
+#include"shell.h"

 #include<assert.h>
+#include<algorithm>
 #include<fcntl.h>
 #include<grp.h>
 #include<pthread.h>
@ -157,15 +159,32 @@ shared_ptr<BusDevice> openRTLWMBUS(Detected detected,
                rtl_wmbus = "rtl_wmbus";
            }
        }
+
        if (command == "")
        {
-            if (!force_freq)
+            string add_f = "";
+            string out;
+            vector<string> args;
+            args.push_back("-h");
+            invokeShellCaptureOutput(rtl_wmbus, { "-h" }, {}, &out, true);
+            debug("(rtlwmbus) help %s\n", out.c_str());
+            if (out.find("-f exit if flow") != string::npos)
            {
-                command = rtl_sdr+" "+ppm+" -d "+to_string(id)+" -f "+freq+" -s 1.6e6 - 2>/dev/null | "+rtl_wmbus+" -s";
+                add_f = " -f";
            }
            else
            {
-                command = rtl_sdr+" "+ppm+" -d "+to_string(id)+" -f "+freq+" -s 1.6e6 - 2>/dev/null | "+rtl_wmbus;
+                warning("Warning! rtl_wbus executable lacks -f option! Without this option rtl_wmbus cannot detect when rtl-sdr stops working.\n"
+                        "Please upgrade rtl_wmbus.\n");
+            }
+
+            if (!force_freq)
+            {
+                command = "ERRFILE=$(mktemp --suffix=_wmbusmeters_rtlsdr) ; echo ERRFILE=$ERRFILE ;  date -Iseconds > $ERRFILE ; tail -f $ERRFILE & "+rtl_sdr+" "+ppm+" -d "+to_string(id)+" -f "+freq+" -s 1.6e6 - 2>>$ERRFILE | "+rtl_wmbus+" -s"+add_f;
+            }
+            else
+            {
+                command = "ERRFILE=$(mktemp --suffix=_wmbusmeters_rtlsdr) ; echo ERRFILE=$ERRFILE ; date -Iseconds > $ERRFILE ; tail -f $ERRFILE & "+rtl_sdr+" "+ppm+" -d "+to_string(id)+" -f "+freq+" -s 1.6e6 - 2>>$ERRFILE | "+rtl_wmbus+" "+add_f;
            }
        }
        verbose("(rtlwmbus) using command: %s\n", command.c_str());
@ -254,6 +273,17 @@ void WMBusRTLWMBUS::processSerialData()
        }
        else if (status == TextAndNotFrame)
        {
+            const char *exit_message = "rtl_wmbus: exiting";
+            auto end = read_buffer_.begin()+frame_length;
+            auto it = std::search(read_buffer_.begin(),
+                                  end,
+                                  exit_message,
+                                  exit_message + strlen(exit_message));
+            if (it != end)
+            {
+                warning("Warning! Detected rtl_wmbus exit due to stopped data flow. Resetting pipeline!\n");
+                reset();
+            }
            // The buffer has already been printed by serial cmd.
            read_buffer_.erase(read_buffer_.begin(), read_buffer_.begin()+frame_length);
        }
--- a/tests/test_bad_driver.sh
+++ b/tests/test_bad_driver.sh
@ -438,6 +438,7 @@ PowerW
 ActualityDuration
 FabricationNo
 EnhancedIdentification
+EnergyMWh
 RelativeHumidity
 AccessNumber
 Medium
--- a/tests/test_t1_meters.sh
+++ b/tests/test_t1_meters.sh
@ -85,6 +85,7 @@ then
        then
            meld $TEST/test_expected.txt $TEST/test_responses.txt
        fi
+        echo ERROR: $TESTNAME
        exit 1
    fi
 else
@ -109,6 +110,7 @@ then
        then
            meld $TEST/test_expected.txt $TEST/test_responses.txt
        fi
+        echo ERROR: $TESTNAME
        exit 1
    fi
 else