refactor: per-patch lifecycle state machine

[eseidel]

Closes shorebirdtech/shorebird#3737.

Why

Per-patch state was scattered across six stores — the download_state.rs sidecar JSON, the bare files in downloads/, the inflated .full file, the patches/{N}/dlc.vmcode artifact, the next_boot_patch / last_booted_patch / currently_booting_patch / known_bad_patches fields of PatchesState, and the session-only running_patch global. There was no single source of truth for "what state is patch N in?"; every behavior change (most recently the 416 fix in #351) added another orthogonal axis instead of moving the patch through defined transitions.

This PR consolidates the persistent per-patch state into a single document at {cache}/patches/{N}/state.json driven by an explicit state machine.

On-disk layout (per release)

{cache}/
  state.json                    # {client_id, release_version, queued_events}
  pointers.json                 # ReleasePointers (patch numbers only)
  patches/
    {N}/
      state.json                # PatchState — source of truth for patch N
      download                  # compressed bytes (Downloading/Downloaded only)
      dlc.vmcode                # installed artifact (Installed only)

Everything under patches/ is wiped on release-version change. Within a release, state.json survives as a Bad tombstone even after artifact files are removed — that's how rollback to a previously-known-bad patch is prevented when newer patches fail.

State machine

enum PatchState {
    Downloading { url, hash, signature, partial_size },
    Downloaded  { url, hash, signature, size },
    Installed   { hash, signature, size },
    Bad         { reason, hash?, signature?, size? },   // tombstone
}

enum BadReason {
    BootCrash,            // boot started but never recorded success
    InvalidPatchBytes,    // inflate failed (zstd magic / decompression)
    InstallHashMismatch,  // bytes decompressed but produced the wrong hash
    ValidationFailed,     // size or signature check failed at boot time
}

pointers.json carries only patch numbers (next_boot_patch, last_booted_patch, currently_booting_patch, boot_started_at). Metadata lives once per patch in its state.json.

API

The lifecycle exposes two state-aware operations callers don't have to choose between:

• mark_bad(n, reason) — writes a Bad tombstone and removes artifact files. Sugar over write_state(Bad{...}) + cleanup.
• cleanup(n) — state-aware retirement. If currently Bad, keeps the tombstone and removes only artifact files. Otherwise removes the entire patch directory.

Plus the lifecycle transitions used by update_internal and report_launch_*:

• decide_start(n, url, hash) → DownloadAction { Fresh, Resume{offset}, Complete, Skip }
• record_download_started, record_download_complete, record_install_complete
• record_boot_start, record_boot_success, record_boot_failure
• detect_boot_crash_on_init (runs on init; promotes a stranded currently_booting_patch to Bad{BootCrash})
• validate_next_boot_patch (size + signature; marks Bad{ValidationFailed} on failure)
• recompute_next_boot (state-aware fallback walk that doesn't clobber a valid newer next_boot_patch)

Behaviors that fall out

• 416 redownload loop (fix: skip fetch when prior download is already complete on disk #351): decide_start returning Complete skips the network when bytes are already on disk; mark_bad(InstallHashMismatch) short-circuits subsequent cycles via Skip(KnownBad). The post-fix: skip fetch when prior download is already complete on disk #351 redownload loop (every cycle: download → hash mismatch → cleanup → repeat) is now bounded to one cycle per (release, patch_number).
• Server rollback un-rollback: uninstall_patch calls cleanup (state-aware), not the old "always-add-to-known-bad" path. A patch the server later un-rolls-back will be redownloaded — but a patch that crashed locally and is then rolled back keeps its Bad{BootCrash} tombstone, so the server's "it's fine again" can't override device-local "this crashed."
• Chunked-encoding resume: decide_start reads the on-disk file size as the resume offset, not a recorded count, so chunked-encoded responses (no Content-Length) work the same as fixed-length ones. The state.json is written before the download starts.
• OS-evicted partial files: decide_start cross-checks the download file's existence for Downloading / Downloaded states. If the file was evicted, the state is unrecoverable — fall through to Fresh.

Diff scope

library/src/cache/lifecycle.rs     | 1365 +++  (new)
library/src/cache/patch_manager.rs | 1615 ---  (deleted)
library/src/download_state.rs      |  126 ---  (deleted)
library/src/cache/updater_state.rs |  622 +-   (rewritten on top of lifecycle)
library/src/updater.rs             |  914 +-   (update_internal cutover, helpers removed)
library/src/cache/mod.rs           |    4 +-
library/src/lib.rs                 |    1 -
7 files changed, 1890 insertions(+), 2757 deletions(-)

Net −867 lines of code despite adding a full state-machine module — the patch_manager / download_state / scattered-state-helpers tonnage outweighs it.

Test plan

• cargo test — 213 tests pass (down from 257 because the patch_manager unit tests are gone with the module). End-to-end coverage in updater.rs::tests is unchanged and all green.
• 67 new lifecycle unit tests covering: serde roundtrip, corrupt state.json fail-safe, mark_bad metadata preservation across all source states, cleanup state-awareness (Bad → tombstone, else → forget), idempotency, decide_start across all states + URL/hash mismatches + missing-file recovery, all download/install transitions, full boot lifecycle (start/success/failure), crash-on-init detection, fallback walk, validation.
• Behavior of every updater::tests resume / rollback / state-recovery / install-failure scenario reproduced end-to-end against the new layout.
• cargo fmt --check clean.

Caveats

• Bumps the on-disk format. Per the design discussion, everything under {cache}/patches/ is wiped on release-version change — no migration needed because per-release state can change freely between releases. Devices upgrading to a release shipped with this PR will redownload any in-flight patch once. This is consistent with how per-release state was always handled.
• config.download_dir is now vestigial. The lifecycle uses {storage_dir}/patches/{N}/ so the prior code_cache_dir/downloads location is no longer used. The field is left in UpdateConfig to keep the C API stable; it can be removed in a follow-up.
• FileOperation::DeleteDir is now only used by tests. Compile-time warning; safe to ignore or remove the variant in a follow-up.

[codecov]

Codecov Report

❌ Patch coverage is 97.46105% with 44 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
library/src/cache/lifecycle.rs	97.98%	24 Missing ⚠️
library/src/cache/updater_state.rs	96.33%	11 Missing ⚠️
library/src/updater.rs	94.97%	9 Missing ⚠️

📢 Thoughts on this report? Let us know!

[eseidel]

@bdero asked for us to stand up end-to-end testing first.

[eseidel]

Built an integration test system per your request: #353

[bdero]

Overall

Right architecture, right time. Consolidating the six scattered state stores into one per-patch state.json driven by an explicit machine is the move, and cleanup being state-aware (Bad → keep tombstone, else → forget entirely) cleanly separates the two prior conflated concerns ("server told us not to use this" vs "this actually crashed on this device"). #356 falls out as a strict subset once this lands — happy to close it.

A handful of suggestions, no blockers. Inline comments tagged 💡 (suggestion) and 🧹 (nit).

💡 No end-to-end test for the customer scenario

library/src/updater.rs doesn't have an end-to-end test that pulls the customer scenario in shorebirdtech/shorebird#3728 through update_internal (install → server rolled_back: [1] → server returns patch 1 again with rolled_back: [] → assert reinstall). The behavior is correct at the lifecycle unit level (cleanup_on_non_bad_patch_forgets_entirely), but the customer reported it through the FFI, so a test at that level would directly nail the regression we just shipped a hotfix for in #356. Cheap to port from #356.

(Inlining this in the body since updater.rs line ~2229 — the natural anchor next to the existing rollback tests — isn't in a diff hunk.)

[bdero]

passes onceover for me, bot found some minor suggestions. Let's merge today and I'll launch a prod smoke test for it.