Skip to content

tools: coverage tooling for block-level aggregation and multi-suite eden runs#5962

Draft
eriknordmark wants to merge 2 commits into
lf-edge:masterfrom
eriknordmark:coverage-tools-doc
Draft

tools: coverage tooling for block-level aggregation and multi-suite eden runs#5962
eriknordmark wants to merge 2 commits into
lf-edge:masterfrom
eriknordmark:coverage-tools-doc

Conversation

@eriknordmark
Copy link
Copy Markdown
Contributor

@eriknordmark eriknordmark commented May 14, 2026

Description

Adds four tools to support EVE's code-coverage workflow, plus matching documentation.

What's new

File Purpose
tools/compute_coverage.sh Computes block-level aggregate coverage across one or more text profiles. Dedupes by (file, range); treats a block as covered if any profile reports a non-zero hit.
tools/eden_run_lib.sh Sourceable Bash library: verify_eve_tag, eden_reset_state, run_eden_suite. Codifies the recurring "SSH-check EVE tag → run suite → reset adam state" pattern.
tools/run_eden_suites.sh Driver using the library. Takes a list of label:dir:scenario specs, runs each with per-suite EVE-tag verification + between-suite reset, optionally collects coverage at the end.
tools/README-coverage.md Co-located README covering the script interfaces, a worked end-to-end example from a multi-PR integration branch (coverage-allprs), and notes for authoring your own driver.
docs/CODE-COVERAGE.md Adds a "Step 5" walkthrough of compute_coverage.sh, a paragraph in "Merging profiles" calling out the double-counting pitfall, and a pointer to the new README.

Why

Three recurring issues in EVE's coverage workflow:

  1. Block-level coverage requires dedup. make coverage-merge concatenates profiles without collapsing identical (file, range) blocks. go tool cover -func aggregates correctly for statement-weighted reporting, but ad-hoc shell counting over the raw file double-counts blocks present in more than one source profile. compute_coverage.sh produces the right block-level number.
  2. Multi-suite Eden runs need drift detection. Several tests (tests/baseosmgr/retry_update.txt, tests/baseosmgr/force_fallback.txt, tests/nodeagent/baseos_fallback_*.txt, tests/update_eve_image/*.txt) deliberately swap EVE's running version and don't always self-clean. Running them back-to-back in one Eden lifetime risks running suite N+1 against a different EVE than expected, invalidating both functional results and coverage. verify_eve_tag after every suite catches this.
  3. Multi-suite drivers were one-off scripts. Today each user building a coverage run hand-rolls the SSH retry loop, the adam reset between suites, and the pre-flight check. Factoring into eden_run_lib.sh makes the pattern reusable.

How to test and validate this PR

compute_coverage.sh

After make test and make eden-cover produce profiles:

tools/compute_coverage.sh "unit"           pkg/pillar/coverage.txt
tools/compute_coverage.sh "unit + eden"    pkg/pillar/coverage.txt \
    dist/amd64/current/eden_coverage/eden_e2e_coverage.txt

Each prints <label> <covered>/<total> = <pct>%. The "unit + eden" covered count should be ≥ "unit" alone (eden can only add coverage, not subtract). For comparison, go tool cover -func=… reports the statement-weighted view; block-level numbers from this script differ by a few percentage points and answer a different question.

run_eden_suites.sh

With Eden up and an onboarded EVE on $EVE_EXPECTED_TAG:

export EDEN=…/eden-linux-amd64
export EDEN_HOME=…/eden_config
export EVE_EXPECTED_TAG=0.0.0-mybranch-deadbeef-kvm-amd64
export EVE_SSH_KEY=…/default-certs/id_rsa
export EDEN_RUNLOGS=…/eden_runlogs

tools/run_eden_suites.sh --assert-no-baseos \
    smoke:./tests/workflow:smoke.tests.txt \
    baseosmgr:./tests/baseosmgr:eden.baseosmgr.tests.txt

Expected behavior:

  • Pre-flight aborts cleanly if $EVE_EXPECTED_TAG doesn't match what's running, or (with --assert-no-baseos) if adam has leftover baseos config.
  • Each suite produces $EDEN_RUNLOGS/<label>.log.
  • A suite that drifts EVE (e.g. retry_update.txt halfway through) triggers an "EVE drifted during this suite — aborting" message from the post-suite verify_eve_tag and exits non-zero.

Help text

tools/compute_coverage.sh --help
tools/run_eden_suites.sh --help

Both print the in-file usage block.

Changelog notes

No user-facing changes. Developer tooling for coverage analysis and multi-suite Eden runs.

PR Backports

  • 16.0-stable: No (developer tooling, not a fix).
  • 14.5-stable: No.
  • 13.4-stable: No.

Checklist

  • I've provided a proper description
  • I've added the proper documentation (docs/CODE-COVERAGE.md + tools/README-coverage.md)
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device — N/A, host-side shell/Python
  • I've written the test verification instructions
  • I've set the proper labels to this PR — to be set

make coverage-merge concatenates the unit / Eden e2e / extras profiles
into combined_coverage.txt without deduplicating. go tool cover -func
handles the duplication correctly for statement-weighted reporting,
but ad-hoc block-level counting over the raw file double-counts blocks
that appear in more than one source profile.

compute_coverage.sh dedupes by (file, range), treats a block as
covered if any input profile recorded a non-zero hit for it, and
prints a single-line summary. Filter is discovered relative to the
script so it works from any workspace. docs/CODE-COVERAGE.md gains a
Step 5 section walking through usage plus a paragraph in "Merging
profiles" calling out the double-counting pitfall.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@github-actions github-actions Bot requested a review from jsfakian May 14, 2026 16:56
@codecov
Copy link
Copy Markdown

codecov Bot commented May 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 21.05%. Comparing base (2caf795) to head (d572d4e).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5962      +/-   ##
==========================================
+ Coverage   20.64%   21.05%   +0.40%     
==========================================
  Files         489      499      +10     
  Lines       90431    92129    +1698     
==========================================
+ Hits        18667    19395     +728     
- Misses      70187    70974     +787     
- Partials     1577     1760     +183     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Driving multiple Eden suites in one Eden lifetime needs three
recurring patterns: SSH-based EVE-tag verification (catches the
retry_update / force_fallback / baseos_fallback_* tests swapping EVE
mid-run), adam state reset between suites, and a pre-flight check
that adam isn't carrying over leftover baseos config from a prior
aborted run.

Factor those into a sourceable library (eden_run_lib.sh) and a
ready-to-use driver (run_eden_suites.sh) that takes a list of
label:dir:scenario specs. Document the end-to-end coverage workflow
— including the worked coverage-allprs example — in
tools/README-coverage.md.

docs/CODE-COVERAGE.md gets a one-paragraph pointer to the new README
so readers landing on the data-flow doc know where the driver scripts
live.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@eriknordmark eriknordmark changed the title tools: compute_coverage.sh for block-level aggregate coverage tools: coverage tooling for block-level aggregation and multi-suite eden runs May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant