tools: coverage tooling for block-level aggregation and multi-suite eden runs#5962
Draft
eriknordmark wants to merge 2 commits into
Draft
tools: coverage tooling for block-level aggregation and multi-suite eden runs#5962eriknordmark wants to merge 2 commits into
eriknordmark wants to merge 2 commits into
Conversation
make coverage-merge concatenates the unit / Eden e2e / extras profiles into combined_coverage.txt without deduplicating. go tool cover -func handles the duplication correctly for statement-weighted reporting, but ad-hoc block-level counting over the raw file double-counts blocks that appear in more than one source profile. compute_coverage.sh dedupes by (file, range), treats a block as covered if any input profile recorded a non-zero hit for it, and prints a single-line summary. Filter is discovered relative to the script so it works from any workspace. docs/CODE-COVERAGE.md gains a Step 5 section walking through usage plus a paragraph in "Merging profiles" calling out the double-counting pitfall. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5962 +/- ##
==========================================
+ Coverage 20.64% 21.05% +0.40%
==========================================
Files 489 499 +10
Lines 90431 92129 +1698
==========================================
+ Hits 18667 19395 +728
- Misses 70187 70974 +787
- Partials 1577 1760 +183 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Driving multiple Eden suites in one Eden lifetime needs three recurring patterns: SSH-based EVE-tag verification (catches the retry_update / force_fallback / baseos_fallback_* tests swapping EVE mid-run), adam state reset between suites, and a pre-flight check that adam isn't carrying over leftover baseos config from a prior aborted run. Factor those into a sourceable library (eden_run_lib.sh) and a ready-to-use driver (run_eden_suites.sh) that takes a list of label:dir:scenario specs. Document the end-to-end coverage workflow — including the worked coverage-allprs example — in tools/README-coverage.md. docs/CODE-COVERAGE.md gets a one-paragraph pointer to the new README so readers landing on the data-flow doc know where the driver scripts live. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds four tools to support EVE's code-coverage workflow, plus matching documentation.
What's new
tools/compute_coverage.sh(file, range); treats a block as covered if any profile reports a non-zero hit.tools/eden_run_lib.shverify_eve_tag,eden_reset_state,run_eden_suite. Codifies the recurring "SSH-check EVE tag → run suite → reset adam state" pattern.tools/run_eden_suites.shlabel:dir:scenariospecs, runs each with per-suite EVE-tag verification + between-suite reset, optionally collects coverage at the end.tools/README-coverage.mddocs/CODE-COVERAGE.mdcompute_coverage.sh, a paragraph in "Merging profiles" calling out the double-counting pitfall, and a pointer to the new README.Why
Three recurring issues in EVE's coverage workflow:
make coverage-mergeconcatenates profiles without collapsing identical(file, range)blocks.go tool cover -funcaggregates correctly for statement-weighted reporting, but ad-hoc shell counting over the raw file double-counts blocks present in more than one source profile.compute_coverage.shproduces the right block-level number.tests/baseosmgr/retry_update.txt,tests/baseosmgr/force_fallback.txt,tests/nodeagent/baseos_fallback_*.txt,tests/update_eve_image/*.txt) deliberately swap EVE's running version and don't always self-clean. Running them back-to-back in one Eden lifetime risks running suite N+1 against a different EVE than expected, invalidating both functional results and coverage.verify_eve_tagafter every suite catches this.eden_run_lib.shmakes the pattern reusable.How to test and validate this PR
compute_coverage.shAfter
make testandmake eden-coverproduce profiles:Each prints
<label> <covered>/<total> = <pct>%. The "unit + eden" covered count should be ≥ "unit" alone (eden can only add coverage, not subtract). For comparison,go tool cover -func=…reports the statement-weighted view; block-level numbers from this script differ by a few percentage points and answer a different question.run_eden_suites.shWith Eden up and an onboarded EVE on
$EVE_EXPECTED_TAG:Expected behavior:
$EVE_EXPECTED_TAGdoesn't match what's running, or (with--assert-no-baseos) if adam has leftover baseos config.$EDEN_RUNLOGS/<label>.log.retry_update.txthalfway through) triggers an "EVE drifted during this suite — aborting" message from the post-suiteverify_eve_tagand exits non-zero.Help text
Both print the in-file usage block.
Changelog notes
No user-facing changes. Developer tooling for coverage analysis and multi-suite Eden runs.
PR Backports
Checklist
docs/CODE-COVERAGE.md+tools/README-coverage.md)