diff --git a/README.md b/README.md index 5f33af3..ee50462 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Z-Wave Controller Backup -**Version:** 1.0.2 | **Author:** CliveS & Claude | **Platform:** Indigo 2025.2 or later, macOS +**Version:** 1.1.0 | **Author:** CliveS & Claude | **Platform:** Indigo 2025.2 or later, macOS Takes a complete copy of your Z-Wave USB controller's memory, checks it, keeps it, and can put it back. Two clicks and about two minutes, from inside Indigo, on the Mac it already runs on. @@ -8,10 +8,10 @@ it back. Two clicks and about two minutes, from inside Indigo, on the Mac it alr ## Why The stick that runs your Z-Wave network keeps the only copy of what that network is: the -Home ID, the list of devices, the routes between them. A 500-series stick can lose that list. -When it does, every device is still on the network and still bound to the same Home ID, but -the stick no longer knows any of them, and Indigo's only documented remedy is to exclude and -include every device again. mat on the Indigo forum had that happen to 106 of them. +Home ID, the list of devices, the routes between them. A stick can lose that list. When it +does, every device is still on the network and still bound to the same Home ID, but the stick +no longer knows any of them, and Indigo's only documented remedy is to exclude and include +every device again. mat on the Indigo forum had that happen to 106 of them. With a copy of the stick's memory on disk the remedy is a two-minute restore. This plugin makes that copy from inside Indigo, needing nothing but the stick you already have. @@ -51,22 +51,35 @@ is when the stick's contents change. The plugin reminds you. ## Which controllers -Any 500-series Z-Wave stick plugged into the Indigo Mac: the Aeotec Z-Stick Gen5 and Gen5+, -the Z-Wave.me UZB, the HomeSeer SmartStick+, the Zooz ZST10 500 and similar. The plugin finds -the port from Indigo's own Z-Wave settings, so there is nothing to configure. +Any 500, 700 or 800 series Z-Wave stick plugged into the Indigo Mac. 500 series: the Aeotec +Z-Stick Gen5 and Gen5+, the Z-Wave.me UZB, the HomeSeer SmartStick+, the Zooz ZST10 500 and +similar. 700 and 800 series: the Zooz ZST10 700 and ZST39, the Aeotec Z-Stick 7 and Z-Stick +10 Pro, the Silicon Labs UZB-7 and similar. The plugin finds the port from Indigo's own Z-Wave +settings, so there is nothing to configure, and works out which generation it is talking to. -Not yet: 700 and 800 series sticks (Z-Stick 7, Z-Stick 10 Pro, ZST39, ZWA-2). Their memory is -read a different way and the author has none to test against. The plugin recognises one and -says so rather than guessing. Sticks connected over the network rather than USB are also out. +The 700 and 800 series keep their memory as a small filesystem rather than a flat block, and +are read and written through a different Serial API command. Two things follow, both explained +in the Event Log when they happen: the stick is reset at the end of every visit (that is the +documented way to leave its memory tidy, and Indigo reconnects to it as normal), and a restore +does not need the unplug-and-replug a 500-series stick needs. A backup takes about twenty +seconds. + +Sticks connected over the network rather than USB are out. ## The one rule about restoring An image only goes back onto the same stick, or another stick of the same model running the same software version. It is a raw copy, and the layout of the memory changed between Aeotec's software versions (an original Gen5 on 1.01 and a Gen5+ on 1.02 are not -interchangeable). The plugin checks this and refuses otherwise. Converting an image for a -different generation of stick is a job for zwave-js, and needs the newer Gen5 software -first; mat's forum write-up (topic 29135) covers that route. +interchangeable), and between the Silicon Labs SDK versions a 700 or 800 series stick runs. +The plugin checks this and refuses otherwise. + +Moving a network onto a different generation of stick (500 to 700, 700 to 800) is a +conversion, not a raw restore, and is a job for zwave-js: its `nvmedit convert` tool, or the +non-raw Restore NVM in Z-Wave JS UI, takes exactly the `.bin` this plugin writes. On a 500 +series stick that route needs the newer Gen5 software first; mat's forum write-up (topic +29135) covers it. Two sticks holding the same Home ID must never be plugged in at the same +time, with or without software driving them. ## Tested @@ -81,6 +94,24 @@ accept a write to the last 16 bytes of its memory. Those bytes are blank in ever so the plugin reads them back instead of forcing the write, confirms they already match, and carries on. It is expected, not a fault. +700 and 800 series, live on 14 September 2026 on a second system: a Zooz ZST39 LR (800 series, +Z-Wave 7.24, ten nodes), an Aeotec Z-Stick 10 Pro (800 series, Z-Wave 7.23, its Z-Wave side; +the Zigbee side is a separate port the plugin ignores) and a Silicon Labs 700 series stick +(Z-Wave 7.17, 108 node ids) running Indigo 2025.2's network. Backups matched a Z-Wave JS UI +backup of the same stick byte for byte, on both of the Serial API commands the plugin uses. A +restore of an older image onto the ZST39 brought back its older node table, and a restore of +the current image put it back; the Z-Stick 10 Pro went through backup, verify, restore and +verify the same way; each restore verified by read-back. + +Two things learnt there, and now built in: a 700/800 stick answers the very last write of a +restore with "end of file" and does not perform it (zwave-js sees the same; those bytes hold +nothing any of its files refer to), and it must not be asked to write those bytes in smaller +pieces, because a one-byte write there hung the ZST39 outright until it was replugged. The +plugin leaves that tail alone and says so. And after any reset the stick may append a few +bytes of its own housekeeping into free space, so a 700/800 read-back is compared everywhere +the image holds data rather than byte for byte, and the node table and Home ID are checked +too. + ## Installation 1. Download the plugin: @@ -91,8 +122,9 @@ carries on. It is expected, not a fault. Backup). It needs no settings. Images go to `Z-Wave Controller Backups` next to the Indigo folder unless you choose another -folder in the plugin's Configure dialog. They are about 256 KB each, so keep them all, and -let your normal backup carry the folder off the Mac. +folder in the plugin's Configure dialog. They are 256 KB each for a 500 series stick and about +40 to 48 KB for a 700 or 800 series one, so keep them all, and let your normal backup carry the +folder off the Mac. ## If something goes wrong @@ -105,6 +137,9 @@ the thread is the one place to look. ## Credits +The 700 and 800 series support was contributed by **Autolog**, tested on a ZST39, a Z-Stick 10 Pro +and the 700 series stick running a second Indigo system, with Claude Opus 5 at the keyboard. + mat's forum write-up of recovering a Gen5 that had lost its node table is what started this, and his zwave-js scripts remain the fallback route for anyone without the plugin. The Serial API details were checked against the zwave-js project's source (MIT). Nothing from either is @@ -112,6 +147,15 @@ bundled here; the plugin is plain Python with no dependencies. ## What's new +**1.1.0** (14-Sep-2026) — 700 and 800 series sticks (Zooz ZST10 700 and ZST39, Aeotec Z-Stick 7 +and Z-Stick 10 Pro, UZB-7 and similar): backup, restore and verify through the NVM3 commands +those sticks use, tested live on a ZST39 LR and a 700 series stick against Z-Wave JS UI. No +unplug needed on this generation; the closing reset does the job. Node ids are read correctly +from a stick that zwave-js has left in 16-bit mode. Show Plugin Info and the device report the +full SDK version. Two small fixes found on the way: a backup folder pasted with quotes around it +is cleaned up rather than landing the images inside the plugin bundle, and the Home ID check +looks at every network Indigo's settings remember instead of the first one it finds. + **1.0.2** (13-Sep-2026) — Show Plugin Info now reports the controller itself: model, Home ID, software version, how many nodes the last image holds, which file it is, and whether the network has changed since. It previously showed only the folder, port and backup time. @@ -125,6 +169,6 @@ with read-back verification, verify, one controller device, and the stale-backup ## Authors & licence -Vibed into existence by **CliveS**, who knew what he wanted, argued until he got it, and tested it on a real house. Typed at inhuman speed by **Claude** (Anthropic), who mostly did as it was told. +Vibed into existence by **CliveS**, who knew what he wanted, argued until he got it, and tested it on a real house. Typed at inhuman speed by **Claude** (Anthropic), who mostly did as it was told. 700 and 800 series by **Autolog** and Claude, on a second real house. © 2026 CliveS · [MIT licence](LICENSE) — copy it, fork it, bend it, break it, fix it, ship it. If it breaks, you get to keep both pieces. diff --git a/SPEC.md b/SPEC.md index 4a0c41b..b20e29e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -219,3 +219,57 @@ Python 3.13 stdlib only (termios/fcntl/json/hashlib), no requirements.txt, versi request (the stick was mid-reply), so the plugin now pauses two seconds, drains and ACKs what the stick still has to say, and backs off on collisions; the Restore dialog's pop-up preselects the newest image. + + +## 700 / 800 series — added in v1.1.0 (14-Sep-2026) + +Verified live on a Zooz ZST39 LR (800 series, "Z-Wave 7.24", SDK 7.24.2, ids 027A:0004:0610, +NVM 40960 bytes) and a Silicon Labs 700 series stick (Z-Wave 7.17, SDK 7.17.1, ids +0000:0004:0004, NVM 49152 bytes, 108 node ids, Indigo 2025.2's live network), against +zwave-js 15.29 whose backupNVMRaw700 serial frames were captured from Z-Wave JS UI and matched. + +**Scope change:** "700/800-series sticks" moves from *Out* to *In*. Conversion between +generations stays out (zwave-js `nvmedit convert`; documented, not done). + +| Use | Function | Notes | +|---|---|---| +| full SDK version ("7.17.1") | 0x09 GetProtocolVersion | payload[1..3]; only asked when the bitmask advertises it | +| function bitmask | 0x07 GetSerialApiCapabilities | bytes 8..39, bit n-1 = function n; decides 0x2E vs 0x3D | +| radio off / on | 0x10 SetRFReceiveMode | [00] before any NVM access, response [01] = done | +| watchdog off | 0xD3 StopWatchdog | no response frame; the closing reset starts it again | +| NVM open/read/write/close | 0x2E NVMOperations | [op, len, offset16]; reply [status, len, offset16, data] | +| the same, 32-bit offsets | 0x3D ExtendedNVMOperations | used when advertised (800 series, NVM over 64 KB); open reply carries a supported-ops bitmask | +| reset | 0x08 SoftReset | mandatory after any NVM visit; the stick answers with an unsolicited 0x0A SerialAPIStarted ~110 ms later | + +Statuses: 0x00 ok, 0xFF end of file (also ok), 0x01 error, 0x02 operation mismatch, 0x03 +operation interference, 0x04 sub-command not supported. Reads ask for 255 bytes and take the +chunk that comes back (64 on both sticks); an empty reply to the probe drops the chunk to 48. +Writes go in the probed chunk. Both sticks answer node ids in 16-bit mode if zwave-js has +put them there, so MemoryGetId and GetSUCNodeId are parsed by payload length. + +**Session:** identity -> radio off -> watchdog off -> open (size) -> close -> [reads / writes] +-> soft reset -> wait for SerialAPIStarted. The reset is always sent on the way out, on failure +too, so the stick is never left with its radio off. No replug on restore: the reset is the +documented step and the read-back follows it at once. + +**Restore facts:** the write whose end reaches the NVM size is answered 0xFF and NOT performed +(zwave-js behaves the same and never checks). Splitting it: 32 bytes at size-64 ok, 32 at +size-32 refused, 16/8/4/2-byte writes down to size-2 ok, then a 1-byte write at size-2 hung +the ZST39 (no ACK, soft reset ignored, watchdog stopped, replug needed). Hence: never retry the +tail in smaller pieces; report it as declined; verify around it. + +**Verification:** byte-identical read-back is not achievable on NVM3 — the firmware appends +housekeeping objects into erased (0xFF) space on restart (74 bytes after one restore) and can +relocate pages after a power cycle. A 700/800 restore is verified by `compare_images_700`: +differences are allowed only inside the declined tail and where the image holds 0xFF; anything +else is "did NOT hold". The Home ID and the node table reported after the reset must equal the +sidecar's. Verify Last Backup uses the same comparison and says how many housekeeping bytes +appeared. + +**Guards added:** `protocolVersionFull` equal when both sides have it (the library string +"Z-Wave 7.17" is coarser than the layout). Known models: ZST39 LR (027A:0004:0610) and the +Silicon Labs 700 reference ids (0000:0004:0004). + +**Tests added:** `FakeStick700` with ZST39 and 700 profiles from the captured payloads, +including the declined tail, the hang on a tail write (asserted never triggered), 16-bit node +ids, the probe fallback, the housekeeping append on reset, and a stick that drops writes. diff --git a/Z-Wave Controller Backup.indigoPlugin/Contents/Info.plist b/Z-Wave Controller Backup.indigoPlugin/Contents/Info.plist index ba7499c..82ffdc1 100644 --- a/Z-Wave Controller Backup.indigoPlugin/Contents/Info.plist +++ b/Z-Wave Controller Backup.indigoPlugin/Contents/Info.plist @@ -25,7 +25,7 @@ Highsteads PluginVersion - 1.0.2 + 1.1.0 ServerApiVersion 3.0 diff --git a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/MenuItems.xml b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/MenuItems.xml index f5bddef..d44d63a 100644 --- a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/MenuItems.xml +++ b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/MenuItems.xml @@ -27,7 +27,7 @@ Restore - + diff --git a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/PluginConfig.xml b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/PluginConfig.xml index 04734fb..f2566f6 100644 --- a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/PluginConfig.xml +++ b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/PluginConfig.xml @@ -4,7 +4,7 @@ - + diff --git a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/controller_image.py b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/controller_image.py index d7b7d87..0dc5de6 100644 --- a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/controller_image.py +++ b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/controller_image.py @@ -6,9 +6,9 @@ # sidecar files and the restore guards. Never imports indigo. Any # object with read() and write() is a port, so everything here runs # under tests against a scripted stick. -# Author: CliveS & Claude Fable 5.1 -# Date: 13-09-2026 12:20 -# Version: 1.0.1 +# Author: CliveS & Claude Fable 5.1; 700/800 series Autolog & Claude Opus 5 +# Date: 14-09-2026 16:30 +# Version: 1.1.0 # # Facts this file rests on (all verified 12-09-2026 against a live Aeotec Gen5, # firmware 1.01, and against @zwave-js/serial 15.29.0): @@ -23,6 +23,32 @@ # stick accepts (zwave-js uses 48 for the Aeotec Gen5 and the Z-Wave.me UZB). # * The write chunk is (largest read reply - 5), the 5 being the offset+length # header, exactly as zwave-js restoreNVMRaw500 does it. +# +# 700 and 800 series (verified 14-09-2026 against a Zooz ZST39 LR, Z-Wave 7.24, and a +# Silicon Labs 700 reference stick, Z-Wave 7.17, and against zwave-js backupNVMRaw700 / +# restoreNVMRaw700 whose serial frames were captured live and matched byte for byte): +# * Their memory is not a flat EEPROM but an NVM3 filesystem, reached through +# NVMOperations 0x2E (open / read / write / close, 16-bit offsets) or, when the +# stick advertises it in its GetSerialApiCapabilities bitmask, ExtendedNVMOperations +# 0x3D (the same with 32-bit offsets, needed once the NVM exceeds 64 KB). Open +# answers with the NVM size; reads asked for 255 bytes come back in whatever chunk +# the stick likes (64 on both sticks here); writes go in that same chunk. +# * The radio is switched off (SetRFReceiveMode 0x10) and the hardware watchdog +# stopped (0xD3) before touching the NVM, and the stick is soft-reset afterwards: +# every 7.x firmware leaves the NVM in an odd state after a read until it restarts +# (zwave-js does the same for the same reason). The reset announces itself with an +# unsolicited SerialAPIStarted 0x0A about 110 ms later, and the radio is on again. +# * The final write, the one that reaches the end of the NVM, is answered with +# "end of file" and NOT written; zwave-js has the same behaviour. Those bytes hold +# nothing any file refers to. They must NOT be retried in smaller pieces: a one-byte +# write near the end hung the ZST39's firmware outright, with the watchdog stopped +# so it could not recover itself, and only a replug brought it back. +# * Byte-identical read-back is not possible on this generation: the firmware appends +# its own housekeeping objects into erased space when it restarts. A restore is +# verified by comparing everywhere the image holds data, plus the identity and node +# table the stick reports afterwards. +# * Node ids come back as two bytes when the stick has been put in 16-bit node id mode +# (zwave-js does that and it survives a soft reset), so they are parsed by length. import datetime import errno @@ -37,7 +63,7 @@ import time import xml.etree.ElementTree as ET -VERSION = "1.0.1" +VERSION = "1.1.0" SOF, ACK, NAK, CAN = 0x01, 0x06, 0x15, 0x18 REQ, RES = 0x00, 0x01 @@ -46,12 +72,26 @@ F_GET_CTRL_CAPS = 0x05 F_GET_SERIALAPI_CAPS = 0x07 F_SOFT_RESET = 0x08 +F_GET_PROTOCOL_VER = 0x09 # 700+: full "7.17.1" version, build number, git hash +F_SERIAL_API_STARTED = 0x0A # unsolicited, sent by a 700+ stick as it comes back from a reset +F_SET_RF_RECEIVE_MODE = 0x10 F_GET_VERSION = 0x15 F_MEMORY_GET_ID = 0x20 F_NVM_GET_ID = 0x29 F_EXT_NVM_READ = 0x2A F_EXT_NVM_WRITE = 0x2B +F_NVM_OPERATIONS = 0x2E # 700+: open/read/write/close, 16-bit offsets +F_EXT_NVM_OPERATIONS = 0x3D # 700+: the same with 32-bit offsets, when advertised F_GET_SUC_NODE_ID = 0x56 +F_STOP_WATCHDOG = 0xD3 # 700+: no response frame + +# NVMOperations / ExtendedNVMOperations sub-commands and statuses (@zwave-js/serial). +NVM_OP_OPEN, NVM_OP_READ, NVM_OP_WRITE, NVM_OP_CLOSE = 0x00, 0x01, 0x02, 0x03 +NVM_ST_OK, NVM_ST_EOF = 0x00, 0xFF +NVM_STATUS_NAMES = {0x00: "ok", 0x01: "error", 0x02: "operation mismatch", 0x03: "operation interference", + 0x04: "sub-command not supported", 0xFF: "end of file"} +NVM700_PROBE = 0xFF # ask for this much; the stick answers in the chunk it likes +NVM700_FALLBACK_CHUNK = 48 # what zwave-js drops to when a 255-byte read comes back empty # NVMSize code (GetNVMId payload[3]) -> bytes, from @zwave-js/serial GetNVMIdMessages. NVM_SIZES = {14: 16 * 1024, 15: 32 * 1024, 16: 64 * 1024, 17: 128 * 1024, 18: 256 * 1024, @@ -79,7 +119,12 @@ KNOWN_MODELS = { (0x0086, 0x0001, 0x005A): "Aeotec Z-Stick Gen5", (0x0115, 0x0400, 0x0001): "Z-Wave.me UZB", + (0x027A, 0x0004, 0x0610): "Zooz ZST39 LR", # 800 series, seen 14-09-2026 + (0x0371, 0x0004, 0x003C): "Aeotec Z-Stick 10 Pro", # 800 series (Z-Wave side of the dual stick), seen 14-09-2026 + (0x0000, 0x0004, 0x0004): "Silicon Labs 700 series stick", # the reference ids many 700 sticks ship with } +# chipType from GetSerialApiInitData -> generation. 5 = 500 (Gen5 measured), 7 and 8 measured 14-09-2026. +CHIP_SERIES = {5: 500, 7: 700, 8: 800} # Sticks that choke on large NVM reads; zwave-js starts these at 48 bytes. CHUNK_48_MODELS = {(0x0086, 0x0001, 0x005A), (0x0115, 0x0400, 0x0001)} # Sticks zwave-js never soft-resets (they do not come back): the replug does the job instead. @@ -390,12 +435,50 @@ def is_500_series(identity): return int(proto.split(".")[0]) < 7 +def is_700_series(identity): + """True for the 700 and 800 series (both speak 'Z-Wave 7.x' and share the NVM3 protocol).""" + proto = protocol_string(identity.get("libraryVersionString")) + return bool(proto) and int(proto.split(".")[0]) >= 7 + + +def series_of(identity): + """500, 700 or 800 from the chip type, falling back to the library string; None if unknown.""" + series = CHIP_SERIES.get(identity.get("chipType")) + if series: + return series + if is_500_series(identity): + return 500 + if is_700_series(identity): + return 700 + return None + + def software_description(identity): lib = identity.get("libraryVersionString") or "unknown" - sdk = sdk_version(lib) + sdk = identity.get("protocolVersionFull") or sdk_version(lib) return f"{lib} (SDK {sdk})" if sdk else lib +def parse_function_bitmask(caps_payload): + """Function ids a stick advertises in GetSerialApiCapabilities (bytes 8..39): bit n-1 set = function n.""" + bm = caps_payload[8:8 + 32] + return [bi * 8 + bit + 1 for bi, by in enumerate(bm) for bit in range(8) if by & (1 << bit)] + + +def node_id_from(payload, start): + """A node id at `start`: one byte, or two big-endian bytes when the stick is in 16-bit + node id mode (the payload is then one byte longer). zwave-js puts 700+ sticks in that + mode and it survives a soft reset, so Indigo's stick can be found either way.""" + if len(payload) >= start + 2: + return int.from_bytes(payload[start:start + 2], "big") + return payload[start] + + +def nvm_function_for(identity): + """Which NVM function a 700+ stick is driven through: 0x3D when it advertises it, else 0x2E.""" + return F_EXT_NVM_OPERATIONS if F_EXT_NVM_OPERATIONS in identity.get("supportedFunctionIds", []) else F_NVM_OPERATIONS + + def read_identity(api): ident = {} p = api.request(F_GET_VERSION) @@ -406,7 +489,7 @@ def read_identity(api): ident["sdkVersion"] = sdk_version(s) p = api.request(F_MEMORY_GET_ID) ident["homeId"] = p[0:4].hex().upper() - ident["ownNodeId"] = p[4] + ident["ownNodeId"] = node_id_from(p, 4) flags = api.request(F_GET_CTRL_CAPS)[0] ident["controllerCapabilityFlags"] = flags ident["isSecondary"] = bool(flags & 0x01) @@ -414,13 +497,20 @@ def read_identity(api): ident["sisPresent"] = bool(flags & 0x04) ident["wasRealPrimary"] = bool(flags & 0x08) ident["isSUC"] = bool(flags & 0x10) - ident["sucNodeId"] = api.request(F_GET_SUC_NODE_ID)[0] + ident["sucNodeId"] = node_id_from(api.request(F_GET_SUC_NODE_ID), 0) p = api.request(F_GET_SERIALAPI_CAPS) ident["serialApiVersion"] = f"{p[0]}.{p[1]}" ident["manufacturerId"] = int.from_bytes(p[2:4], "big") ident["productType"] = int.from_bytes(p[4:6], "big") ident["productId"] = int.from_bytes(p[6:8], "big") ident["modelName"] = model_name(ident["manufacturerId"], ident["productType"], ident["productId"]) + ident["supportedFunctionIds"] = parse_function_bitmask(p) + ident["protocolVersionFull"] = None + if is_700_series(ident) and F_GET_PROTOCOL_VER in ident["supportedFunctionIds"]: + # [protocol type, major, minor, patch, build (2), git hash (16)]; only the version matters here + p = api.request(F_GET_PROTOCOL_VER) + if len(p) >= 4: + ident["protocolVersionFull"] = f"{p[1]}.{p[2]}.{p[3]}" p = api.request(F_GET_INIT_DATA) ident["initApiVersion"] = p[0] caps = p[1] @@ -544,6 +634,193 @@ def soft_reset(api): return api.send(F_SOFT_RESET) +# --------------------------------------------------------------------------- +# 700 and 800 series: NVM3 through NVMOperations (0x2E) / ExtendedNVMOperations (0x3D) +# --------------------------------------------------------------------------- + +def radio_off(api): + """Silence the radio so the protocol cannot write to the NVM while it is being copied.""" + resp = api.request(F_SET_RF_RECEIVE_MODE, b"\x00") + return bool(resp) and resp[0] == 1 + + +def radio_on(api): + resp = api.request(F_SET_RF_RECEIVE_MODE, b"\x01") + return bool(resp) and resp[0] == 1 + + +def stop_watchdog(api): + """The hardware watchdog would reset the stick mid-transfer; it has no response frame.""" + return api.send(F_STOP_WATCHDOG) + + +def begin_nvm_session_700(api): + """Radio off and watchdog stopped, as zwave-js does before any NVM access. Raises if + the radio will not go quiet: copying a live NVM is not worth having.""" + if not radio_off(api): + raise RuntimeError("the controller would not switch its radio off before the memory access") + stop_watchdog(api) + + +def end_nvm_session_700(api, timeout=8.0): + """Soft reset, then wait for the stick to say it is back (SerialAPIStarted). The reset + is what puts the NVM right again after a read and turns the radio and watchdog back + on; there is no other way out of the session. Returns True once the stick announced + itself, False if it merely went quiet (a replug then does the same job).""" + if not api.send(F_SOFT_RESET): + return False + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + kind, fr = api._next_event(min(0.5, max(0.0, deadline - time.monotonic()))) + if kind == "frame" and fr["func"] == F_SERIAL_API_STARTED: + return True + return False + + +class Nvm700: + """One NVM3 conversation with a 700+ stick over `func` (0x2E or 0x3D). The offset is two + bytes on 0x2E and four on 0x3D, in both the request and the response.""" + + def __init__(self, api, func=F_EXT_NVM_OPERATIONS): + self.api = api + self.func = func + self.offset_width = 4 if func == F_EXT_NVM_OPERATIONS else 2 + + def _call(self, op, payload=b""): + resp = self.api.request(self.func, bytes([op]) + payload) + if len(resp) < 2: + raise RuntimeError(f"NVM operation {op} got a {len(resp)}-byte reply") + status, dlen = resp[0], resp[1] + w = self.offset_width + offset = int.from_bytes(resp[2:2 + w], "big") if len(resp) >= 2 + w else 0 + data = resp[2 + w:2 + w + dlen] + if status not in (NVM_ST_OK, NVM_ST_EOF): + raise RuntimeError(f"NVM operation {op} failed: {NVM_STATUS_NAMES.get(status, status)}") + return status, offset, data + + def open(self): + """Returns the NVM size in bytes.""" + _status, size, _data = self._call(NVM_OP_OPEN) + if size <= 0: + raise RuntimeError("the controller reported an empty NVM") + return size + + def read(self, offset, length): + return self._call(NVM_OP_READ, bytes([length]) + offset.to_bytes(self.offset_width, "big")) + + def write(self, offset, piece): + return self._call(NVM_OP_WRITE, bytes([len(piece)]) + offset.to_bytes(self.offset_width, "big") + piece) + + def close(self): + self._call(NVM_OP_CLOSE) + + +def read_nvm_id_700(api, func): + """The 700+ counterpart of read_nvm_id: open the NVM for its size and close it again.""" + nvm = Nvm700(api, func) + size = nvm.open() + nvm.close() + return {"sizeBytes": size, "function": f"0x{func:02X}", "nvmManufacturerId": None, "memoryType": "NVM3"} + + +def dump_nvm_700(api, func, size, progress=None, should_stop=None): + """Read the whole NVM3 area, exactly as zwave-js backupNVMRaw700: ask for 255 bytes, take + whatever chunk the stick answers with, drop to 48 if it answers with nothing.""" + nvm = Nvm700(api, func) + opened = nvm.open() + if opened != size: + nvm.close() + raise RuntimeError(f"the NVM is {opened} bytes now but was {size} bytes a moment ago") + data = bytearray() + chunk = min(NVM700_PROBE, size) + next_pct = 10 + try: + while len(data) < size: + if should_stop and should_stop(): + raise RuntimeError("stopped") + status, offset, piece = nvm.read(len(data), min(chunk, size - len(data))) + if not piece: + if chunk == NVM700_PROBE: + chunk = NVM700_FALLBACK_CHUNK + continue + raise RuntimeError(f"NVM read at offset {len(data)} returned nothing") + if offset != len(data): + raise RuntimeError(f"NVM read came back for offset {offset}, not {len(data)}") + data += piece + if chunk > len(piece): + chunk = len(piece) + if progress and len(data) * 100 // size >= next_pct: + progress(len(data), size) + next_pct += 10 + if status == NVM_ST_EOF: + break + finally: + nvm.close() + if len(data) != size: + raise RuntimeError(f"read {len(data)} of {size} bytes before the controller said end of file") + return bytes(data) + + +def write_nvm_700(api, func, image, progress=None, should_stop=None): + """Write `image` back, exactly as zwave-js restoreNVMRaw700: probe the chunk with one read, + then write chunk by chunk. Returns (bytes written, bytes the controller declined at the + end). The final chunk, the one that reaches the end of the NVM, is answered 'end of + file' and is not written; that is how these sticks behave and it is left alone. It is + NEVER split and retried: a one-byte write at the end hung a ZST39 (14-09-2026).""" + nvm = Nvm700(api, func) + size = nvm.open() + if size != len(image): + nvm.close() + raise RuntimeError(f"the image is {len(image)} bytes but the controller's NVM is {size}") + _s, _o, probe = nvm.read(0, min(NVM700_PROBE, size)) + chunk = len(probe) or NVM700_FALLBACK_CHUNK + nvm.close() + nvm.open() + offset = 0 + declined = 0 + next_pct = 10 + try: + while offset < len(image): + if should_stop and should_stop(): + raise RuntimeError("stopped") + piece = image[offset:offset + chunk] + status, _echo, _d = nvm.write(offset, piece) + if status == NVM_ST_EOF: + declined = len(image) - offset + offset = len(image) + break + offset += len(piece) + if progress and offset * 100 // len(image) >= next_pct: + progress(offset, len(image)) + next_pct += 10 + finally: + nvm.close() + return offset - declined, declined + + +def compare_images_700(image, current, declined_tail=0): + """Compare a 700+ image with a later read of the same stick, allowing for what this + generation does on its own: it appends housekeeping objects into erased (0xFF) space when + it restarts, and it never writes the final chunk of a restore. Returns the plain diff plus + how many differing bytes fall into each of those two allowances; `unexplained` is what is + left, and that is the number that decides whether a restore held.""" + diff = compare_images(image, current) + tail_start = len(image) - declined_tail + housekeeping = tail = unexplained = 0 + for i, (x, y) in enumerate(zip(image, current)): + if x == y: + continue + if i >= tail_start: + tail += 1 + elif x == 0xFF: + housekeeping += 1 + else: + unexplained += 1 + unexplained += abs(len(image) - len(current)) + diff.update(housekeepingBytes=housekeeping, tailBytes=tail, unexplainedBytes=unexplained) + return diff + + # --------------------------------------------------------------------------- # Image checks, sidecars, guards # --------------------------------------------------------------------------- @@ -652,6 +929,12 @@ def restore_refusals(image, sidecar, identity, nvm, replacement_ok=False): reasons.append( f"the image came from software '{ident_img.get('libraryVersionString')}' and this controller runs " f"'{identity.get('libraryVersionString')}'; the memory layout differs between versions") + elif ident_img.get("protocolVersionFull") and identity.get("protocolVersionFull") \ + and ident_img["protocolVersionFull"] != identity["protocolVersionFull"]: + # 700+ sticks say "7.17" in the library string and "7.17.1" in full; the full one is the layout + reasons.append( + f"the image came from SDK {ident_img['protocolVersionFull']} and this controller runs " + f"SDK {identity['protocolVersionFull']}; the memory layout can differ between those") if ident_img.get("homeId") != identity.get("homeId") and not replacement_ok: reasons.append( f"the image is for Home ID {ident_img.get('homeId')} and this controller is {identity.get('homeId')}; " @@ -663,6 +946,28 @@ def restore_refusals(image, sidecar, identity, nvm, replacement_ok=False): # Indigo's own Z-Wave settings (read, never written) # --------------------------------------------------------------------------- +def clean_folder_path(text): + """A folder path as a person typed or pasted it: surrounding quotes and whitespace dropped, + `~` expanded. Returns "" for blank, and for anything that is not an absolute path, because + a relative path would land the images inside the plugin bundle (seen 14-09-2026 with a + path pasted in shell quotes).""" + folder = (text or "").strip().strip("'\"").strip() + if not folder: + return "" + folder = os.path.expanduser(folder) + return folder if os.path.isabs(folder) else "" + + +def zwave_home_ids_from_prefs(pref_path): + """Every Home ID Indigo's Z-Wave prefs mention. Old networks linger there for years, so + the stick's Home ID is checked against the set, never against the first one found.""" + try: + root = ET.parse(pref_path).getroot() + except (OSError, ET.ParseError): + return set() + return {m.group(1) for el in root.iter() for m in [re.fullmatch(r"Home([0-9A-F]{8})_Node\d+", el.tag or "")] if m} + + def zwave_port_from_prefs(pref_path): """(port, connection_type, home_id) from Indigo's Z-Wave prefs file, or (None, None, None).""" try: diff --git a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/plugin.py b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/plugin.py index 1fe2a22..18da99c 100644 --- a/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/plugin.py +++ b/Z-Wave Controller Backup.indigoPlugin/Contents/Server Plugin/plugin.py @@ -2,13 +2,13 @@ # -*- coding: utf-8 -*- # Filename: plugin.py # Description: Z-Wave Controller Backup. Takes a complete, verified image of a -# 500-series Z-Wave USB controller's memory (Home ID, node table, -# routes) and can write it back. Indigo has to let go of the stick -# while that happens, so the plugin waits for the user to switch -# Z-Wave off, does the job, and says when to switch it back on. -# Author: CliveS & Claude Fable 5.1 -# Date: 13-09-2026 17:45 -# Version: 1.0.2 +# 500, 700 or 800 series Z-Wave USB controller's memory (Home ID, +# node table, routes) and can write it back. Indigo has to let go of +# the stick while that happens, so the plugin waits for the user to +# switch Z-Wave off, does the job, and says when to switch it back on. +# Author: CliveS & Claude Fable 5.1; 700/800 series Autolog & Claude Opus 5 +# Date: 14-09-2026 16:30 +# Version: 1.1.0 try: import indigo @@ -41,7 +41,7 @@ def as_bool(value, default=False): # ============================================================ PLUGIN_ID = "com.clives.indigoplugin.zwave-controller-backup" -PLUGIN_VERSION = "1.0.2" +PLUGIN_VERSION = "1.1.0" DEVICE_TYPE = "zwaveController" DEFAULT_FOLDER_NAME = "Z-Wave Controller Backups" POLL_SECONDS = 2 @@ -63,8 +63,10 @@ def as_bool(value, default=False): versions. Fallback without the plugin: the zwave-js library's `controller.restoreNVMRaw()` writes the -same bytes back (never `restoreNVM()`, which tries to convert). See mat's write-up on the -Indigo forum, topic 29135, for a ready-made script. +same bytes back. See mat's write-up on the Indigo forum, topic 29135, for a ready-made +script. Moving a network onto a different generation of stick (500 to 700, 700 to 800) is +a conversion, not a raw restore: zwave-js's `restoreNVM()` or its `nvmedit convert` tool +does that, and these images are exactly the input it takes. Take a fresh copy whenever a device is added to or removed from the network. """ @@ -90,6 +92,7 @@ def __init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs): self._network_flag = False self._final_status = None self._shadow = {} # last states written, for when no device exists yet + self._session_open = False # a 700/800 stick has its radio off and needs the closing reset self._read_prefs(pluginPrefs) # -------------------------------------------------------- @@ -97,7 +100,10 @@ def __init__(self, pluginId, pluginDisplayName, pluginVersion, pluginPrefs): # -------------------------------------------------------- def _read_prefs(self, prefs): - folder = (prefs.get("backupFolder") or "").strip() + raw = prefs.get("backupFolder") or "" + folder = ci.clean_folder_path(raw) + if raw.strip() and not folder: + self.logger.warning(f"The backup folder setting '{raw.strip()}' is not a full path; using the default folder instead.") self.backup_folder = folder or self._default_folder() self.port_override = (prefs.get("portOverride") or "").strip() try: @@ -133,8 +139,12 @@ def validatePrefsConfigUi(self, valuesDict): raise ValueError except (ValueError, TypeError): errors["waitMinutes"] = "Whole minutes between 1 and 60." - folder = (valuesDict.get("backupFolder") or "").strip() - if folder: + raw = (valuesDict.get("backupFolder") or "").strip() + folder = ci.clean_folder_path(raw) + if raw and not folder: + errors["backupFolder"] = "Enter the full path of a folder, starting with / (or ~), without quotes." + elif folder: + valuesDict["backupFolder"] = folder # keep what was meant, not the quotes around it try: _os.makedirs(folder, exist_ok=True) except OSError as e: @@ -531,22 +541,56 @@ def _with_stick(self, name, body): self.logger.info(f"Z-Wave is off. Opening {port_path}.") self._set(actionRequired=False, backupStatus=f"{name.capitalize()} in progress") port, api = self._open(port_path) + self._session_open = False try: identity = ci.read_identity(api) - if not ci.is_500_series(identity): + if ci.is_500_series(identity): + nvm = ci.read_nvm_id(api) + elif ci.is_700_series(identity): + # 700/800: radio off + watchdog stopped for the whole visit, soft reset on the way out. + self._session_begin(api) + nvm = ci.read_nvm_id_700(api, ci.nvm_function_for(identity)) + else: self.logger.error( - f"This controller reports '{identity.get('libraryVersionString')}', which is a 700 or 800 series stick. " - f"Version {PLUGIN_VERSION} images 500-series controllers only.") + f"This controller reports '{identity.get('libraryVersionString')}', which the plugin does not recognise " + f"as a 500, 700 or 800 series stick.") self._final_status = f"{name.capitalize()} failed" return False - nvm = ci.read_nvm_id(api) return body(port_path, port, api, identity, nvm) finally: + try: + self._session_end(api) + except Exception as e: + self.logger.debug(f"could not reset the controller on the way out: {e}") try: port.close() except Exception: pass + def _session_begin(self, api): + ci.begin_nvm_session_700(api) + self._session_open = True + + def _session_end(self, api): + """Soft-reset a 700/800 stick (radio and watchdog come back with it). No-op otherwise.""" + if not self._session_open: + return + self._session_open = False + if not ci.end_nvm_session_700(api): + self.logger.warning("The controller did not announce itself after the reset. If Indigo cannot reconnect, unplug and replug the stick.") + + @staticmethod + def _series(identity): + return ci.series_of(identity) or 500 + + def _dump(self, api, identity, nvm, label): + """One full read of the controller, whichever generation it is.""" + if self._series(identity) >= 700: + return ci.dump_nvm_700(api, ci.nvm_function_for(identity), nvm["sizeBytes"], + progress=self._progress(label), should_stop=self._stop.is_set) + return ci.dump_nvm(api, nvm["sizeBytes"], ci.initial_chunk_for(identity), + progress=self._progress(label), should_stop=self._stop.is_set) + def _progress(self, label): def cb(done, total): self.logger.debug(f"{label}: {done * 100 // total}% ({done}/{total} bytes)") @@ -587,17 +631,20 @@ def _run_backup(self): def _backup_body(self, port_path, port, api, identity, nvm): size = nvm["sizeBytes"] - _p, _c, prefs_home = ci.zwave_port_from_prefs(self._zwave_prefs_path()) - if prefs_home and prefs_home != identity["homeId"]: - self.logger.warning(f"The stick's Home ID is {identity['homeId']} but Indigo's Z-Wave settings name {prefs_home}. Imaging the stick as it is.") + known_homes = ci.zwave_home_ids_from_prefs(self._zwave_prefs_path()) + if known_homes and identity["homeId"] not in known_homes: + self.logger.warning( + f"The stick's Home ID is {identity['homeId']} but Indigo's Z-Wave settings only know " + f"{', '.join(sorted(known_homes))}. Imaging the stick as it is.") nodes = [n for n in identity["nodeIds"] if n != identity["ownNodeId"]] + series = self._series(identity) + how_long = "about twenty seconds" if series >= 700 else "about a minute" self.logger.info( - f"Reading {identity['modelName']}, Home ID {identity['homeId']}, {ci.software_description(identity)}, " - f"{len(nodes)} nodes, {size // 1024} KB of memory. Two full reads, about a minute.") - chunk = ci.initial_chunk_for(identity) + f"Reading {identity['modelName']} ({series} series), Home ID {identity['homeId']}, {ci.software_description(identity)}, " + f"{len(nodes)} nodes, {size // 1024} KB of memory. Two full reads, {how_long}.") t0 = time.monotonic() - img1 = ci.dump_nvm(api, size, chunk, progress=self._progress("read 1"), should_stop=self._stop.is_set) - img2 = ci.dump_nvm(api, size, chunk, progress=self._progress("read 2"), should_stop=self._stop.is_set) + img1 = self._dump(api, identity, nvm, "read 1") + img2 = self._dump(api, identity, nvm, "read 2") diff = ci.compare_images(img1, img2) checks, problems = ci.image_checks(img1, size, identity["homeId"]) if diff["differingBytes"]: @@ -640,9 +687,28 @@ def _verify_body(self, port_path, port, api, identity, nvm): with open(self._last_path, "rb") as fh: image = fh.read() self.logger.info(f"Reading the controller to compare with {_os.path.basename(self._last_path)}.") - current = ci.dump_nvm(api, nvm["sizeBytes"], ci.initial_chunk_for(identity), - progress=self._progress("verify read"), should_stop=self._stop.is_set) - diff = ci.compare_images(image, current) + current = self._dump(api, identity, nvm, "verify read") + if self._series(identity) >= 700: + # NVM3 only ever appends: routes the stick learns and its housekeeping land in erased space, and + # nothing the image holds is overwritten until the stick compacts a page. So free-space additions + # are normal; an overwritten byte, or a node table that differs, is a real change. + diff = ci.compare_images_700(image, current) + image_nodes = sorted(self._last.get("identity", {}).get("nodeIds", [])) + if diff["unexplainedBytes"] == 0 and sorted(identity.get("nodeIds", [])) == image_nodes: + note = (f" ({diff['housekeepingBytes']} bytes have been added in free space since: routes the controller " + f"has learnt and its own housekeeping, which this generation writes on its own; the node table is " + f"the same and nothing the image holds has been overwritten)" + if diff["differingBytes"] else " byte for byte") + self.logger.info(f"Verified: the controller's memory matches the last backup{note}.") + self._final_status = f"Backed up {self._pretty(self._last['takenAt'])}, verified" + self._set(lastResult="verify ok: matches the last backup") + return True + if sorted(identity.get("nodeIds", [])) != image_nodes: + diff["differingBytes"] = max(diff["differingBytes"], 1) + self.logger.warning( + f"The controller now lists {len(identity.get('nodeIds', []))} nodes where the last backup has {len(image_nodes)}.") + else: + diff = ci.compare_images(image, current) if diff["differingBytes"] == 0: self.logger.info("Verified: the controller's memory matches the last backup byte for byte.") self._final_status = f"Backed up {self._pretty(self._last['takenAt'])}, verified" @@ -673,6 +739,8 @@ def _restore_body(self, port_path, port, api, identity, nvm, path, replacement_o self._final_status = "Restore refused" self._set(lastResult="restore refused: " + "; ".join(reasons)) return False + if self._series(identity) >= 700: + return self._restore_body_700(port_path, port, api, identity, nvm, path, image, side) self.logger.info( f"Writing {_os.path.basename(path)} ({len(image)} bytes) into {identity['modelName']}, Home ID {identity['homeId']}. " f"Do not unplug anything yet.") @@ -720,6 +788,53 @@ def _restore_body(self, port_path, port, api, identity, nvm, path, replacement_o self._set(lastResult=f"restore failed: read-back differs in {diff['differingBytes']} bytes") return False + def _restore_body_700(self, port_path, port, api, identity, nvm, path, image, side): + """700/800: write, soft reset (no unplug needed, the reset is the documented step), read + back straight away and compare with the two allowances this generation needs.""" + func = ci.nvm_function_for(identity) + self.logger.info( + f"Writing {_os.path.basename(path)} ({len(image)} bytes) into {identity['modelName']}, Home ID {identity['homeId']}. " + f"Do not unplug anything.") + written, declined = ci.write_nvm_700(api, func, image, progress=self._progress("write"), should_stop=self._stop.is_set) + if declined: + self.logger.info( + f"The controller answered 'end of file' to the final {declined} bytes and did not write them, which is how " + f"this generation behaves (zwave-js sees the same). Those bytes hold nothing any of its files refer to; " + f"the plugin leaves them alone rather than forcing them, which is known to hang the stick.") + self.logger.info(f"Written {written} bytes. Resetting the controller and reading it back.") + self._session_end(api) # soft reset: the stick reloads its NVM3 files from what was written + self._session_begin(api) + current = self._dump(api, identity, nvm, "read-back") + ident2 = ci.read_identity(api) + diff = ci.compare_images_700(image, current, declined) + want_nodes = sorted(side["identity"].get("nodeIds", [])) + got_nodes = sorted(ident2.get("nodeIds", [])) + if diff["unexplainedBytes"] == 0 and ident2.get("homeId") == side["identity"].get("homeId") and got_nodes == want_nodes: + extra = [] + if diff["housekeepingBytes"]: + extra.append(f"{diff['housekeepingBytes']} bytes of housekeeping the controller wrote into free space on restart") + if diff["tailBytes"]: + extra.append(f"the {diff['tailBytes']} unwritten bytes at the end") + note = f" (the only differences are {' and '.join(extra)})" if extra else " byte for byte" + self.logger.info( + f"Restore verified: everything the image holds is in the controller{note}, Home ID {ident2['homeId']}, " + f"{len([n for n in got_nodes if n != ident2['ownNodeId']])} nodes as in the image.") + self._final_status = f"Restored {self._pretty(side.get('takenAt', ''))}, verified" + self._set(lastResult="restore ok: read-back matches the image", **self._identity_states(ident2)) + return True + why = [] + if diff["unexplainedBytes"]: + why.append(f"the read-back differs from the image in {diff['unexplainedBytes']} bytes that are not housekeeping (first difference at offset {diff['firstDifference']})") + if ident2.get("homeId") != side["identity"].get("homeId"): + why.append(f"the Home ID is now {ident2.get('homeId')}, not {side['identity'].get('homeId')}") + if got_nodes != want_nodes: + why.append(f"the controller lists {len(got_nodes)} nodes where the image has {len(want_nodes)}") + self.logger.error( + f"Restore did NOT hold: {'; '.join(why)}. Do not switch Z-Wave back on yet. Run Restore again, or restore your previous image.") + self._final_status = "Restore failed" + self._set(lastResult="restore failed: " + "; ".join(why)) + return False + def _wait_for_replug(self, port_path): """Wait for the port node to vanish (unplug) and return (replug). Carry on either way.""" t0 = time.monotonic() diff --git a/tests/fake_stick.py b/tests/fake_stick.py index 3504135..f83676c 100644 --- a/tests/fake_stick.py +++ b/tests/fake_stick.py @@ -1,15 +1,18 @@ #! /usr/bin/env python # -*- coding: utf-8 -*- # Filename: fake_stick.py -# Description: A scripted 500-series controller behind a read()/write() port, for -# the contract tests. Answers the Serial API functions the plugin -# uses with payloads shaped exactly like the live Aeotec Gen5's -# (captured 12-09-2026), serves NVM reads from a bytearray, applies -# writes to it, and can inject unsolicited frames, a corrupt frame, -# a CAN, or empty NVM replies to exercise the parser's rough edges. -# Author: CliveS & Claude Fable 5.1 -# Date: 13-09-2026 12:30 -# Version: 1.0.0 +# Description: Scripted controllers behind a read()/write() port, for the contract +# tests. FakeStick answers the Serial API functions the plugin uses +# with payloads shaped exactly like the live Aeotec Gen5's (captured +# 12-09-2026), serves NVM reads from a bytearray, applies writes to +# it, and can inject unsolicited frames, a corrupt frame, a CAN, or +# empty NVM replies to exercise the parser's rough edges. FakeStick700 +# does the same for the 700/800 series with payloads captured from a +# Zooz ZST39 LR and a Silicon Labs 700 stick (14-09-2026), including +# the unwritten final chunk and the hang a write into the tail causes. +# Author: CliveS & Claude Fable 5.1; 700/800 series Autolog & Claude Opus 5 +# Date: 14-09-2026 16:30 +# Version: 1.1.0 import importlib.util import os @@ -189,3 +192,165 @@ def _respond(self, func, payload): if func == ci.F_SOFT_RESET: return None return b"" + + +# ---------------------------------------------------------------- 700 / 800 series + +# Payloads captured 14-09-2026. ZST39: GetVersion "Z-Wave 7.24" type 7, GetSerialApiCapabilities +# 01 46 027A 0004 0610 + bitmask (advertises 0x2E, 0x3D and 0x09), GetSerialApiInitData api 10, +# GetProtocolVersion 00 07 18 02 + build + git hash. The 700 stick is the same shape with +# "Z-Wave 7.17", ids 0000:0004:0004, api 9, and no 0x3D in its bitmask. +ZST39 = { + "library": b"Z-Wave 7.24\x00" + bytes([7]), + "homeId": bytes.fromhex("E2DAD17B"), "ownNode": 1, + "caps": 0x18, "suc": 1, + "serialapi": bytes.fromhex("0146027a00040610f6873e88cf2bc05fe3d7fde0970f008000808680ba0d00f00000ee7fc0000000"), + "protocol": bytes.fromhex("00071802abcd") + b"8bcd01f6b2a24ecc", + "init_api": 10, + "nodes": [1, 2, 3, 4, 5, 6, 7, 9, 11, 15], + "chip": (8, 0), + "nvm_size": 40960, +} +STICK700 = { + "library": b"Z-Wave 7.17\x00" + bytes([7]), + "homeId": bytes.fromhex("E1C3BDE8"), "ownNode": 1, + "caps": 0x1C, "suc": 1, + "serialapi": bytes.fromhex("0711000000040004f6873e88cf2bc04ffbd7fde00700008000808680ba05007000002e7fc0000000"), + "protocol": bytes.fromhex("00071101abcd") + b"0123456789abcdef", + "init_api": 9, + "nodes": [1, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 38, 92, 93, 105, 106, 111, 114, 116, 120, 128], + "chip": (7, 0), + "nvm_size": 49152, +} + + +class FakeStick700(FakeStick): + """A port whose other end is a 700 or 800 series controller. NVM3 is served through + 0x2E and, when the profile advertises it, 0x3D, in `chunk`-byte replies (64 on both + real sticks). The write that reaches the end of the NVM is answered 'end of file' and + dropped, as the real ones do; a write ending inside the last two bytes hangs the stick + (no ACK, ever), as the ZST39 did. A soft reset is answered, after the ACK, with an + unsolicited SerialAPIStarted, and it turns the radio back on and appends a small + housekeeping object into the first erased space, as the firmware does on restart.""" + + SERIAL_API_STARTED = bytes.fromhex("0700010100085e989f556c568f7401031d0000") # captured from the ZST39 + + def __init__(self, nvm=None, profile=ZST39, chunk=64, node_id_bits=8, housekeeping=b"\xd7\x00\x8a\x8a\xfe\x00\x00\x00\x03\x06\x00\x00\x00\x03"): + super().__init__(nvm=nvm if nvm is not None else self.default_nvm700(profile), profile=profile) + self.chunk = chunk + self.node_id_bits = node_id_bits + self.housekeeping = housekeeping + self.radio_on = True + self.watchdog_on = True + self.nvm_open = False + self.hung = False + self.resets = 0 + self.tail_writes_declined = 0 + self.empty_probe_replies = 0 # answer this many 255-byte reads with nothing (SDK quirk) + + @staticmethod + def default_nvm700(profile): + """Something NVM3-shaped enough for the checks: page headers, objects with the Home ID + in them, erased space at the end of each 8 KB page, and a data tail like the ZST39's.""" + size = profile["nvm_size"] + data = bytearray(b"\xff" * size) + page = 8192 + for p in range(size // page): + base = p * page + data[base:base + 20] = bytes.fromhex("01009ab20c0000c8f3ffff17ffffffff3990ffff") + for i in range(20, page // 2): + data[base + i] = (i * 13 + p) & 0xFF + data[base + 64:base + 68] = profile["homeId"] + data[size - 64:size] = bytes.fromhex("0000000000000253bc01400bfd05") + bytes(50) + return data + + def _node_id(self, n): + return n.to_bytes(2, "big") if self.node_id_bits == 16 else bytes([n]) + + def _pump(self): + if self.hung: + self.inbox.clear() + return + super()._pump() + + def _respond(self, func, payload): + p = self.profile + if func == ci.F_MEMORY_GET_ID: + return p["homeId"] + self._node_id(p["ownNode"]) + if func == ci.F_GET_SUC_NODE_ID: + return self._node_id(p["suc"]) + if func == ci.F_GET_INIT_DATA: + return bytes([p["init_api"], 0b1000, 29]) + node_bitmask(p["nodes"]) + bytes(p["chip"]) + if func == ci.F_GET_PROTOCOL_VER: + return p["protocol"] + if func == ci.F_SET_RF_RECEIVE_MODE: + self.radio_on = bool(payload[0]) + return bytes([1]) + if func == ci.F_STOP_WATCHDOG: + self.watchdog_on = False + return None + if func == ci.F_SOFT_RESET: + self.resets += 1 + self.radio_on = self.watchdog_on = True + self.nvm_open = False + self._append_housekeeping() + self.outbox += request_frame(ci.F_SERIAL_API_STARTED, self.SERIAL_API_STARTED) + return None + if func in (ci.F_NVM_OPERATIONS, ci.F_EXT_NVM_OPERATIONS): + return self._nvm_op(func, payload) + if func in (ci.F_NVM_GET_ID, ci.F_EXT_NVM_READ, ci.F_EXT_NVM_WRITE): + return b"" # a 700 stick has none of the 500-series NVM functions + return super()._respond(func, payload) + + def _append_housekeeping(self): + """What the firmware does on restart: a new object in the first erased run it finds.""" + if not self.housekeeping: + return + run = bytes(b"\xff" * (len(self.housekeeping) + 8)) + at = bytes(self.nvm).find(run, 8192) + if at >= 0: + self.nvm[at:at + len(self.housekeeping)] = self.housekeeping + + def _nvm_op(self, func, payload): + if func == ci.F_EXT_NVM_OPERATIONS and 0x3D not in ci.parse_function_bitmask(self.profile["serialapi"]): + return bytes([0x04, 0]) # sub-command not supported + w = 4 if func == ci.F_EXT_NVM_OPERATIONS else 2 + op = payload[0] + size = len(self.nvm) + + def reply(status, offset_or_size, data=b""): + out = bytes([status, len(data)]) + offset_or_size.to_bytes(w, "big") + data + if op == ci.NVM_OP_OPEN and func == ci.F_EXT_NVM_OPERATIONS: + out = bytes([status, 1]) + offset_or_size.to_bytes(w, "big") + bytes([0x0F]) + return out + + if op == ci.NVM_OP_OPEN: + self.nvm_open = True + return reply(ci.NVM_ST_OK, size) + if op == ci.NVM_OP_CLOSE: + self.nvm_open = False + return reply(ci.NVM_ST_OK, 0) + if not self.nvm_open: + return reply(0x02, 0) # operation mismatch + length = payload[1] + offset = int.from_bytes(payload[2:2 + w], "big") + if op == ci.NVM_OP_READ: + if length == 0xFF and self.empty_probe_replies: + self.empty_probe_replies -= 1 + return reply(ci.NVM_ST_OK, offset) + n = min(length, self.chunk, size - offset) + data = bytes(self.nvm[offset:offset + n]) + return reply(ci.NVM_ST_EOF if offset + n >= size else ci.NVM_ST_OK, offset, data) + if op == ci.NVM_OP_WRITE: + data = payload[2 + w:2 + w + length] + end = offset + len(data) + if end >= size: + self.tail_writes_declined += 1 + return reply(ci.NVM_ST_EOF, offset) # the last chunk: not written + if end > size - 2 or offset >= size - 2: + self.hung = True # what the ZST39 did to a 1-byte write at size-2 + return None + self.nvm[offset:end] = data + self.write_count += 1 + return reply(ci.NVM_ST_OK, offset) + return reply(0x04, 0) diff --git a/tests/test_controller_image.py b/tests/test_controller_image.py index be44bb3..3e2a423 100644 --- a/tests/test_controller_image.py +++ b/tests/test_controller_image.py @@ -6,8 +6,8 @@ # traffic, identity from Gen5-shaped payloads, adaptive reads, the # write path, image checks, sidecars and every restore guard. # Author: CliveS & Claude Fable 5.1 -# Date: 13-09-2026 12:35 -# Version: 1.0.0 +# Date: 14-09-2026 16:30 +# Version: 1.1.0 import datetime import json @@ -15,7 +15,7 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from fake_stick import GEN5, FakeStick, ci, request_frame, response_frame # noqa: E402 +from fake_stick import GEN5, STICK700, ZST39, FakeStick, FakeStick700, ci, request_frame, response_frame # noqa: E402 def api_for(stick): @@ -305,6 +305,30 @@ def test_zwave_port_and_home_id_read_from_indigo_prefs(tmp_path): assert ci.zwave_port_from_prefs(os.path.join(tmp_path, "missing")) == (None, None, None) +def test_every_home_id_in_indigo_prefs_is_known_not_just_the_first(tmp_path): + """A real prefs file (14-09-2026) carried five Home IDs from past networks; the first was + not the live one, and the plugin warned about a mismatch that was not there.""" + pref = os.path.join(tmp_path, "zwave.indiPref") + with open(pref, "w") as fh: + fh.write('' + 'true' + 'true' + 'true' + '') + assert ci.zwave_home_ids_from_prefs(pref) == {"0161EC56", "E1C3BDE8"} + assert ci.zwave_home_ids_from_prefs(os.path.join(tmp_path, "missing")) == set() + + +def test_folder_path_is_cleaned_and_must_be_absolute(): + """Pasted in shell quotes, a relative path put the images inside the plugin bundle (14-09-2026).""" + assert ci.clean_folder_path("'/Users/me/Z-Wave Backups'") == "/Users/me/Z-Wave Backups" + assert ci.clean_folder_path(' "/Users/me/x" ') == "/Users/me/x" + assert ci.clean_folder_path("~/x").startswith("/") and ci.clean_folder_path("~/x").endswith("/x") + assert ci.clean_folder_path("relative/path") == "" + assert ci.clean_folder_path("'relative'") == "" + assert ci.clean_folder_path("") == "" and ci.clean_folder_path(None) == "" + + def test_port_holders_is_empty_when_lsof_is_unavailable(monkeypatch): monkeypatch.setattr(ci, "LSOF", "/nonexistent/lsof") assert ci.port_holders("/dev/cu.nothing") == [] @@ -422,3 +446,160 @@ def _respond(self, func, payload): assert "2 attempts" in str(e) and "no reply" in str(e) else: raise AssertionError("must raise") + + +# ---------------------------------------------------------------- 700 / 800 series + +def test_identity_from_zst39_shaped_payloads(): + ident = ci.read_identity(api_for(FakeStick700())) + assert ident["libraryVersionString"] == "Z-Wave 7.24" + assert ident["protocolVersionFull"] == "7.24.2" + assert ident["homeId"] == "E2DAD17B" and ident["ownNodeId"] == 1 and ident["sucNodeId"] == 1 + assert ident["modelName"] == "Zooz ZST39 LR" + assert (ident["chipType"], ident["chipVersion"]) == (8, 0) + assert ident["nodeIds"] == ZST39["nodes"] + assert 0x2E in ident["supportedFunctionIds"] and 0x3D in ident["supportedFunctionIds"] + assert not ci.is_500_series(ident) and ci.is_700_series(ident) + assert ci.series_of(ident) == 800 + assert ci.nvm_function_for(ident) == ci.F_EXT_NVM_OPERATIONS + assert ci.software_description(ident) == "Z-Wave 7.24 (SDK 7.24.2)" + + +def test_identity_from_a_700_stick_uses_the_16_bit_function(): + ident = ci.read_identity(api_for(FakeStick700(profile=STICK700))) + assert ident["libraryVersionString"] == "Z-Wave 7.17" and ident["protocolVersionFull"] == "7.17.1" + assert ci.series_of(ident) == 700 + assert 0x3D not in ident["supportedFunctionIds"] + assert ci.nvm_function_for(ident) == ci.F_NVM_OPERATIONS + assert ident["modelName"] == "Silicon Labs 700 series stick" + + +def test_node_ids_are_parsed_by_length_in_16_bit_mode(): + """zwave-js leaves 700+ sticks in 16-bit node id mode; MemoryGetId then answers 6 bytes.""" + ident = ci.read_identity(api_for(FakeStick700(node_id_bits=16))) + assert ident["ownNodeId"] == 1 and ident["sucNodeId"] == 1 + assert ci.node_id_from(bytes.fromhex("e2dad17b01"), 4) == 1 + assert ci.node_id_from(bytes.fromhex("e2dad17b0001"), 4) == 1 + assert ci.node_id_from(bytes.fromhex("e2dad17b0102"), 4) == 0x0102 + + +def test_function_bitmask_bit_n_minus_1_is_function_n(): + caps = bytes(8) + bytes([0b00000001, 0b10000000]) + bytes(30) + assert ci.parse_function_bitmask(caps) == [1, 16] + + +def test_500_series_identity_is_unchanged_by_the_new_fields(): + ident = ci.read_identity(api_for(FakeStick())) + assert ident["protocolVersionFull"] is None and ident["supportedFunctionIds"] == [] + assert ci.series_of(ident) == 500 and not ci.is_700_series(ident) + + +def test_nvm_session_turns_the_radio_off_and_the_reset_brings_the_stick_back(): + stick = FakeStick700() + api = api_for(stick) + ci.begin_nvm_session_700(api) + assert not stick.radio_on and not stick.watchdog_on + assert ci.end_nvm_session_700(api) is True # SerialAPIStarted seen + assert stick.radio_on and stick.watchdog_on and stick.resets == 1 + + +def test_dump_700_reads_the_whole_nvm_in_the_sticks_chunk_via_0x3d(): + stick = FakeStick700() + api = api_for(stick) + size = ci.read_nvm_id_700(api, ci.F_EXT_NVM_OPERATIONS)["sizeBytes"] + assert size == 40960 + img = ci.dump_nvm_700(api, ci.F_EXT_NVM_OPERATIONS, size) + assert img == bytes(stick.nvm) + reads = [pl for f, pl in stick.requests if f == ci.F_EXT_NVM_OPERATIONS and pl[0] == ci.NVM_OP_READ] + assert reads[0][1] == 0xFF and reads[1][1] == 64 # probe, then the stick's chunk + assert stick.requests[-1] == (ci.F_EXT_NVM_OPERATIONS, bytes([ci.NVM_OP_CLOSE])) + + +def test_dump_700_via_0x2e_uses_two_byte_offsets(): + stick = FakeStick700(profile=STICK700) + api = api_for(stick) + size = ci.read_nvm_id_700(api, ci.F_NVM_OPERATIONS)["sizeBytes"] + img = ci.dump_nvm_700(api, ci.F_NVM_OPERATIONS, size) + assert img == bytes(stick.nvm) and len(img) == 49152 + reads = [pl for f, pl in stick.requests if f == ci.F_NVM_OPERATIONS and pl[0] == ci.NVM_OP_READ] + assert len(reads[1]) == 1 + 1 + 2 and int.from_bytes(reads[1][2:4], "big") == 64 + + +def test_dump_700_falls_back_to_48_when_the_probe_comes_back_empty(): + stick = FakeStick700() + stick.empty_probe_replies = 1 + api = api_for(stick) + img = ci.dump_nvm_700(api, ci.F_EXT_NVM_OPERATIONS, 40960) + assert img == bytes(stick.nvm) + reads = [pl for f, pl in stick.requests if f == ci.F_EXT_NVM_OPERATIONS and pl[0] == ci.NVM_OP_READ] + assert reads[1][1] == 48 + + +def test_dump_700_refuses_a_size_that_moved(): + import pytest + with pytest.raises(RuntimeError): + ci.dump_nvm_700(api_for(FakeStick700()), ci.F_EXT_NVM_OPERATIONS, 12345) + + +def test_write_700_lands_every_byte_but_the_declined_tail_and_never_touches_the_last_two(): + stick = FakeStick700() + original = bytes(stick.nvm) + image = bytearray(original) + for i in range(20, 4000): + image[i] ^= 0x5A + api = api_for(stick) + written, declined = ci.write_nvm_700(api, ci.F_EXT_NVM_OPERATIONS, bytes(image)) + assert (written, declined) == (40960 - 64, 64) + assert stick.tail_writes_declined == 1 + assert not stick.hung # no split-and-retry into the tail + assert bytes(stick.nvm[:-64]) == bytes(image[:-64]) + assert bytes(stick.nvm[-64:]) == original[-64:] # the tail is what it was + writes = [pl for f, pl in stick.requests if f == ci.F_EXT_NVM_OPERATIONS and pl[0] == ci.NVM_OP_WRITE] + assert all(pl[1] == 64 for pl in writes) + + +def test_write_700_refuses_an_image_of_the_wrong_size(): + import pytest + with pytest.raises(RuntimeError): + ci.write_nvm_700(api_for(FakeStick700()), ci.F_EXT_NVM_OPERATIONS, b"\xff" * 100) + + +def test_a_write_into_the_tail_would_hang_the_fake_as_it_hung_the_real_stick(): + """Documents the hazard the write path is built to avoid.""" + stick = FakeStick700() + nvm = ci.Nvm700(api_for(stick)) + nvm.open() + import pytest + with pytest.raises(RuntimeError): + nvm.write(40960 - 2, b"\x00") + assert stick.hung + + +def test_compare_700_explains_housekeeping_and_tail_but_not_real_differences(): + image = bytes([0x11] * 100 + [0xFF] * 50 + [0x22] * 64) + current = bytearray(image) + current[110:120] = b"\xd7" * 10 # the firmware wrote into erased space + current[-64:] = b"\xff" * 64 # the declined tail + d = ci.compare_images_700(image, bytes(current), declined_tail=64) + assert (d["housekeepingBytes"], d["tailBytes"], d["unexplainedBytes"]) == (10, 64, 0) + current[5] = 0x00 # a real difference + d = ci.compare_images_700(image, bytes(current), declined_tail=64) + assert d["unexplainedBytes"] == 1 and d["firstDifference"] == 5 + assert ci.compare_images_700(image, image)["differingBytes"] == 0 + + +def test_restore_guard_refuses_a_different_full_sdk_on_the_same_library_string(tmp_path): + ident = ci.read_identity(api_for(FakeStick700())) + nvm = {"sizeBytes": 40960} + image = b"\xff" * 40960 + side = {"identity": dict(ident)} + assert ci.restore_refusals(image, side, ident, nvm) == [] + side["identity"]["protocolVersionFull"] = "7.19.0" + reasons = ci.restore_refusals(image, side, ident, nvm) + assert len(reasons) == 1 and "7.19.0" in reasons[0] + + +def test_image_checks_pass_on_an_nvm3_shaped_image(): + stick = FakeStick700() + checks, problems = ci.image_checks(bytes(stick.nvm), 40960, "E2DAD17B") + assert problems == [] and checks["homeIdOccurrences"] >= 1 diff --git a/tests/test_plugin_smoke.py b/tests/test_plugin_smoke.py index c295fed..495a7f7 100644 --- a/tests/test_plugin_smoke.py +++ b/tests/test_plugin_smoke.py @@ -9,8 +9,8 @@ # a port while Z-Wave is on, and Show Plugin Info describes the # controller from the newest image. # Author: CliveS & Claude Fable 5.1 -# Date: 13-09-2026 17:50 -# Version: 1.1.0 +# Date: 14-09-2026 16:30 +# Version: 1.2.0 import importlib.util import json @@ -23,7 +23,7 @@ import pytest sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from fake_stick import GEN5, SERVER_PLUGIN, FakeStick # noqa: E402 +from fake_stick import GEN5, SERVER_PLUGIN, STICK700, ZST39, FakeStick, FakeStick700 # noqa: E402 # ── the smallest Indigo that will hold still ───────────────────────── @@ -167,6 +167,20 @@ def test_plugin_imports_starts_and_writes_the_folder_readme(tmp_path): assert plugin.wait_minutes == 1 +def test_backup_warns_only_when_the_home_id_is_unknown_to_indigo(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + wire_stick(mod, plugin, FakeStick()) # Home ID E0BB7FA8, which the stub prefs know + run_and_join(plugin, plugin.menuBackup, {}) + assert not any("Home ID" in m for m in spy.messages(logging.WARNING)) + wire_stick(mod, plugin, FakeStick700()) # E2DAD17B, which they do not + run_and_join(plugin, plugin.menuBackup, {}) + assert any("only know E0BB7FA8" in m for m in spy.messages(logging.WARNING)) + + def test_backup_end_to_end_writes_image_and_sidecar_and_states(tmp_path): zw = {"on": False} mod, plugin, ind = load_plugin(tmp_path, zw) @@ -375,6 +389,14 @@ def test_prefs_validation_and_menu_prefill(tmp_path): assert ok is False and "waitMinutes" in errors ok, values = plugin.validatePrefsConfigUi({"waitMinutes": "5", "backupFolder": os.path.join(str(tmp_path), "elsewhere")})[:2] assert ok is True and os.path.isdir(os.path.join(str(tmp_path), "elsewhere")) + # quotes around a pasted path are dropped, and a relative path is refused rather than landing in the bundle + quoted = "'" + os.path.join(str(tmp_path), "quoted") + "'" + ok, values = plugin.validatePrefsConfigUi({"waitMinutes": "5", "backupFolder": quoted})[:2] + assert ok is True and values["backupFolder"] == os.path.join(str(tmp_path), "quoted") and os.path.isdir(values["backupFolder"]) + ok, values, errors = plugin.validatePrefsConfigUi({"waitMinutes": "5", "backupFolder": "relative/folder"}) + assert ok is False and "backupFolder" in errors and not os.path.exists("relative/folder") + plugin._read_prefs({"backupFolder": "'relative'", "waitMinutes": "5"}) + assert plugin.backup_folder == plugin._default_folder() assert plugin.get_menu_action_config_ui_values("menuBackup")["folderShown"] == plugin.backup_folder assert plugin.imageList()[0][1].startswith("No images") assert "imageFile" not in plugin.get_menu_action_config_ui_values("menuRestore") @@ -395,20 +417,171 @@ def test_restore_dialog_preselects_the_newest_image(tmp_path): assert plugin.get_menu_action_config_ui_values("menuRestore")["imageFile"] == newest -def test_700_series_is_refused_before_any_read(tmp_path): +def test_unrecognised_generation_is_refused_before_any_read(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + stick = FakeStick(profile=dict(GEN5, library=b"Z-Wave\x00" + bytes([1]))) # no version at all + wire_stick(mod, plugin, stick) + run_and_join(plugin, plugin.menuBackup, {}) + assert any("does not recognise" in m for m in spy.messages(logging.ERROR)) + assert not any(f in (mod.ci.F_EXT_NVM_READ, mod.ci.F_NVM_OPERATIONS, mod.ci.F_EXT_NVM_OPERATIONS) for f, _ in stick.requests) + assert mod.ci.list_images(plugin.backup_folder) == [] + + +@pytest.mark.parametrize("profile,func,size", [(ZST39, 0x3D, 40960), (STICK700, 0x2E, 49152)]) +def test_700_series_backup_end_to_end(tmp_path, profile, func, size): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + stick = FakeStick700(profile=profile) + wire_stick(mod, plugin, stick) + before = bytes(stick.nvm) + run_and_join(plugin, plugin.menuBackup, {}) + path, side = mod.ci.list_images(plugin.backup_folder)[0] + assert open(path, "rb").read() == before + assert side["nvm"]["sizeBytes"] == size and side["nvm"]["function"] == f"0x{func:02X}" + assert side["identity"]["homeId"] == profile["homeId"].hex().upper() + # radio off and watchdog stopped before the first NVM op, reset on the way out, radio back on + funcs = [f for f, _ in stick.requests] + assert funcs.index(mod.ci.F_SET_RF_RECEIVE_MODE) < funcs.index(func) + assert funcs.index(mod.ci.F_STOP_WATCHDOG) < funcs.index(func) + assert funcs[-1] == mod.ci.F_SOFT_RESET and stick.resets == 1 and stick.radio_on + assert not any(f in (mod.ci.F_NVM_GET_ID, mod.ci.F_EXT_NVM_READ) for f in funcs) # no 500-series calls + assert any("Backup complete" in m for m in spy.messages(logging.INFO)) + assert any(f"({700 if func == 0x2E else 800} series)" in m for m in spy.messages(logging.INFO)) + st = plugin._shadow + assert st["softwareVersion"].startswith("Z-Wave 7.") and st["homeId"] == side["identity"]["homeId"] + + +def test_700_series_restore_end_to_end_without_a_replug(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + stick = FakeStick700() + wire_stick(mod, plugin, stick) + replug_calls = [] + plugin._wait_for_replug = lambda port_path: replug_calls.append(port_path) + run_and_join(plugin, plugin.menuBackup, {}) + path, side = mod.ci.list_images(plugin.backup_folder)[0] + image = open(path, "rb").read() + # scribble on the stick so the restore has work to do + stick.nvm[100:2000] = b"\x00" * 1900 + writes_before = stick.write_count + run_and_join(plugin, plugin.menuRestore, {"imageFile": path, "confirmOverwrite": "true"}) + assert stick.write_count > writes_before + assert replug_calls == [] # the reset is the step, not an unplug + assert bytes(stick.nvm[100:2000]) == image[100:2000] # the scribble is gone + d = mod.ci.compare_images_700(image, bytes(stick.nvm), 64) # only housekeeping + tail may differ + assert d["unexplainedBytes"] == 0 + assert any("Restore verified" in m for m in spy.messages(logging.INFO)) + assert any("end of file" in m and "did not write them" in m for m in spy.messages(logging.INFO)) + assert stick.resets == 3 and not stick.hung and stick.radio_on # backup 1, restore 2: after the write and on the way out + assert plugin._shadow["lastResult"].startswith("restore ok") + + +def test_700_series_restore_notices_a_write_that_did_not_take(tmp_path): zw = {"on": False} mod, plugin, ind = load_plugin(tmp_path, zw) spy = LogSpy() plugin.logger.addHandler(spy) plugin.startup() - stick = FakeStick(profile=dict(GEN5, library=b"Z-Wave 7.19\x00" + bytes([1]))) + stick = FakeStick700() wire_stick(mod, plugin, stick) run_and_join(plugin, plugin.menuBackup, {}) - assert any("700 or 800 series" in m for m in spy.messages(logging.ERROR)) - assert not any(f == mod.ci.F_EXT_NVM_READ for f, _ in stick.requests) + path, side = mod.ci.list_images(plugin.backup_folder)[0] + + class Forgetful(FakeStick700): + """Says OK to every write and keeps none of it.""" + def _nvm_op(self, func, payload): + if payload[0] == mod.ci.NVM_OP_WRITE and self.nvm_open: + w = 4 if func == mod.ci.F_EXT_NVM_OPERATIONS else 2 + return bytes([0, 0]) + payload[2:2 + w] + return super()._nvm_op(func, payload) + forget = Forgetful() + forget.nvm[100:2000] = b"\x00" * 1900 + wire_stick(mod, plugin, forget) + run_and_join(plugin, plugin.menuRestore, {"imageFile": path, "confirmOverwrite": "true"}) + assert any("did NOT hold" in m and "not housekeeping" in m for m in spy.messages(logging.ERROR)) + assert plugin._shadow["lastResult"].startswith("restore failed") + + +def test_700_series_restore_notices_a_node_table_that_differs(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + stick = FakeStick700() + wire_stick(mod, plugin, stick) + run_and_join(plugin, plugin.menuBackup, {}) + path, side = mod.ci.list_images(plugin.backup_folder)[0] + stick.profile["nodes"] = ZST39["nodes"] + [40] # the stick keeps reporting a node the image lacks + run_and_join(plugin, plugin.menuRestore, {"imageFile": path, "confirmOverwrite": "true"}) + assert any("did NOT hold" in m and "lists 11 nodes where the image has 10" in m for m in spy.messages(logging.ERROR)) + + +def test_700_series_verify_tolerates_housekeeping_and_reports_real_change(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + stick = FakeStick700() + wire_stick(mod, plugin, stick) + run_and_join(plugin, plugin.menuBackup, {}) + # the reset at the end of the backup already appended housekeeping into erased space + run_and_join(plugin, plugin.menuVerify) + assert any("Verified" in m and "added in free space" in m for m in spy.messages(logging.INFO)) + stick.profile["nodes"] = ZST39["nodes"] + [40] # a node appeared, though every byte of it landed in free space + run_and_join(plugin, plugin.menuVerify) + assert any("lists 11 nodes where the last backup has 10" in m for m in spy.messages(logging.WARNING)) + assert plugin._shadow["networkChangedSinceBackup"] is True + stick.profile["nodes"] = ZST39["nodes"] + plugin._network_flag = False + stick.nvm[200:210] = b"\x00" * 10 # a real change to a node file + run_and_join(plugin, plugin.menuVerify) + assert any("differs from the last backup" in m for m in spy.messages(logging.WARNING)) + + +def test_700_series_session_is_closed_with_a_reset_even_when_the_read_fails(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + spy = LogSpy() + plugin.logger.addHandler(spy) + plugin.startup() + + class Broken(FakeStick700): + def _nvm_op(self, func, payload): + if payload[0] == mod.ci.NVM_OP_READ and self.nvm_open: + return bytes([0x03, 0]) # operation interference, every time + return super()._nvm_op(func, payload) + stick = Broken() + wire_stick(mod, plugin, stick) + run_and_join(plugin, plugin.menuBackup, {}) + assert any("Backup failed" in m and "interference" in m for m in spy.messages(logging.ERROR)) + assert stick.resets == 1 and stick.radio_on # never left with the radio off assert mod.ci.list_images(plugin.backup_folder) == [] +def test_16_bit_node_id_mode_does_not_confuse_the_backup(tmp_path): + zw = {"on": False} + mod, plugin, ind = load_plugin(tmp_path, zw) + plugin.startup() + stick = FakeStick700(node_id_bits=16) + wire_stick(mod, plugin, stick) + run_and_join(plugin, plugin.menuBackup, {}) + path, side = mod.ci.list_images(plugin.backup_folder)[0] + assert side["identity"]["ownNodeId"] == 1 and side["identity"]["sucNodeId"] == 1 + assert plugin._shadow["nodeCount"] == len(ZST39["nodes"]) - 1 + + @pytest.mark.parametrize("conn", ["netSocket", "netRfc2217"]) def test_network_attached_interface_is_refused(tmp_path, conn): zw = {"on": False}