tests/nodeagent: e2e coverage for reboot paths, restart counter, and disk-space maintenance

[eriknordmark]

Summary

Adds a new tests/nodeagent/ suite with six e2e tests that exercise nodeagent's safety-net reboot paths, restart-counter persistence, and the maintenance-mode-on-no-disk-space chain.

Cloud-disconnect reboot paths

Each in two distinct network-failure modes:

Test	Path tested	Failure mode
reset_on_disconnect_link_down	BootReasonDisconnect	eth0 link-down (eden eve link down / QEMU set_link off)
reset_on_disconnect_blackhole	BootReasonDisconnect	controller IP silently dropped via in-EVE iptables -A OUTPUT -d <adam-ip> -j DROP; link, IP and lease all stay valid
baseos_fallback_link_down	BootReasonFallback	eth0 link-down during the post-update test window
baseos_fallback_blackhole	BootReasonFallback	controller IP black-holed during the test window

The two failure modes exercise different code paths: link-down trips NIM's DPC verification first; the black-hole keeps the interface healthy so nodeagent's reset / fallback timer is the primary defence. The black-hole case is closer to common real-world failures (ISP routing change, controller IP migration, upstream firewall) than yanking the cable.

Restart counter + disk-space maintenance

Test	Path tested
restart_counter_monotonic	RestartCounter increments by exactly +1 across an actual EVE reboot, end-to-end through /persist/status/restartcounter → NodeAgentStatus → zedagent info → controller
maintenance_no_disk_space	RemainingSpace=0 (driven by a blank volume + filler outside AppPersistPaths) propagates through volumemgr → nodeagent → zedagent → controller as MaintenanceMode:true + MaintenanceModeReasons:[LOW_DISK_SPACE], and clears cleanly when pressure is removed

Test plan

• All six tests pass locally against a coverage-instrumented EVE under QEMU (ZedVirtual-4G).
• Each black-hole rule is wiped naturally on the post-disconnect reboot — no cleanup needed.
• Disk-space maintenance test cleans up filler + volume; device returns to ZDEVICE_STATE_ONLINE.
• CI run on lf-edge/eden's harness.

Per-test runtime (laptop, KVM-accelerated): 4–8 min each.

Implementation notes

• Freshness guard for reboot tests: each reset_on_disconnect_* prefixes with eden controller edge-node reboot to establish a known LastBootReason baseline. The final lim.test match thus cannot succeed against stale info from a prior run. The baseos_fallback_* tests get the same freshness for free since the upgrade flow always transits BootReasonUpdate before reaching BootReasonFallback.
• Info publish cadence: timer.deviceinfo.interval=30 so the rebooted device publishes a fresh info message quickly. lim.test's TestInfo matches only new info (einfo.InfoNew), so without this it can wait up to 10 min for the next periodic push.
• Config propagation timing: timer.config.interval=10 shortens the controller-config polling. The fallback tests then exec sleep 70 so the new shorter interval has time to take effect under the default 60 s polling, before any subsequent config push.
• Idempotency across runs (fallback tests): eveimage-remove + brief wait before eveimage-update, so re-running with the same EVE version still triggers a fresh update. (EVE refuses to retry a previously-failed version unless the retry counter is bumped or the baseos config is removed first.)
• Black-hole mechanism: eden eve ssh -- iptables -A OUTPUT -d <adam-ip> -j DROP. The -- terminates eden's cobra flag parsing so the iptables flags pass through cleanly. Empirically verified the rule lands in pillar/zedbox's network namespace (it shares net:[4026531840] with the dom0 shell), so it does silently drop zedagent's controller-bound traffic.
• Disk-pressure mechanism: blank volume at 60 % of /persist (sparse — declared size feeds reservedAppDiskUsage, on-disk cost is small) + 70 % filler in /persist/log/ to push dynamicUsedByDom0 past the 2 GiB Dom0DiskUsageMaxBytes static cap. The filler MUST NOT go to /persist/newlog/ — that directory is in volumemgr's excludeDirs and so cancels out in usedByDom0 = device.Used - appUsage. timer.metric.diskscan.interval=30 makes volumemgr's recompute window a tractable test duration.
• Disk-pressure cleanup: the volume-create / fill / assert sequence runs inside a bash wrapper invoked via exec -t 30m bash run_test.sh, with trap cleanup EXIT tearing down the multi-GiB filler and the blank volume regardless of which assertion fails. The escript dialect has no defer of its own, so without the wrapper a flaky MaintenanceMode:true match (info-publish lag, propagation timeout) would halt the script with the filler still pinned to disk and wedge /persist for any subsequent run.
• Device-model gating: tests skip with a message line on device models other than ZedVirtual-4G / VBox (no eden eve link / eden eve ssh support).

Suite structure

tests/nodeagent/
├── Makefile
├── eden-config.yml
├── eden.nodeagent.tests.txt
└── testdata/
    ├── baseos_fallback_blackhole.txt
    ├── baseos_fallback_link_down.txt
    ├── maintenance_no_disk_space.txt
    ├── reset_on_disconnect_blackhole.txt
    ├── reset_on_disconnect_link_down.txt
    └── restart_counter_monotonic.txt

The suite is not wired into tests/workflow/smoke.tests.txt or eve-upgrade.tests.txt in this PR — the six tests are addressable on their own (e.g. eden.escript.test -test.run TestEdenScripts/reset_on_disconnect_link_down -testdata tests/nodeagent/testdata/). Folding them into a broader suite is a follow-up after we see how they behave under CI.

Why draft

Want a chance for someone familiar with eden's harness to look at the patterns before marking ready — particularly the per-test timer tuning, the idempotency dance for the fallback tests, and the blank-volume-plus-filler trick for the disk-space test.