Skip to content

ci: collect EVE coverage per matrix job in Eden CI#1173

Open
eriknordmark wants to merge 2 commits into
lf-edge:masterfrom
eriknordmark:ci-coverage-collection
Open

ci: collect EVE coverage per matrix job in Eden CI#1173
eriknordmark wants to merge 2 commits into
lf-edge:masterfrom
eriknordmark:ci-coverage-collection

Conversation

@eriknordmark
Copy link
Copy Markdown
Contributor

@eriknordmark eriknordmark commented May 12, 2026

Description

Per-job coverage collection for the Eden test matrix plus a fan-in
merge job, intended to be consumed by a coverage-instrumented EVE
build on the lf-edge/eve side.

Each suite in the Eden test matrix expands into ~12 independent
runners with their own ./eden setup -> start -> stop -> make clean
lifecycle, so per-job collection is the only point at which to
capture covdata. After make clean the runner is wiped — there is no
later opportunity.

Two commits:

collect-coverage: preserve raw covdata in output

CollectEveCoverage SCP'd the raw covmeta.* / covcounters.*
files into a temp dir, ran go tool covdata textfmt on them, and
let the deferred cleanup wipe the binary files on return. Only the
text profile at <output-dir>/eden_e2e_coverage.txt survived.

Callers that want to merge coverage across multiple runs (e.g.
across the Eden test matrix) need the raw covdata so they can use
go tool covdata merge; the text profile alone only supports
concatenation, which is fine for a per-line summary but doesn't
aggregate the way merge does.

Copy the SCP target into <output-dir>/covdata/ instead of an
auto-deleted temp dir. The text-profile output path is unchanged.

ci: collect eden e2e coverage per matrix job

  • run-eden-test composite action gains a Collect/Upload coverage
    pair, gated as best-effort (if: always() and || true), that
    invokes ./eden eve collect-coverage and uploads the resulting
    eden_coverage/ directory (text profile + raw covdata subtree) as
    artifact covdata-<suite>-fs-<fs>-tpm-<tpm>-attempt-<n>.
  • test.yml gains a coverage-merge job that downloads every
    covdata-* artifact, runs go tool covdata merge on the raw
    covmeta.* / covcounters.* files to produce a single merged covdata
    directory, converts it to a text profile via covdata textfmt,
    and runs go tool cover -func for a per-package summary.

Companion PR

This change is the prerequisite for
lf-edge/eve#5950,
which adds an opt-in COVER=y EVE build (gated on a coverage PR
label) and wires eden-trusted.yml to consume the resulting
artifact. The EVE PR calls
lf-edge/eden/.github/workflows/test.yml@master, so this PR has to
land first; otherwise the tests-master-cover job on the EVE side
will run tests but emit no covdata-merged artifact.

How to test and validate this PR

Without a coverage-instrumented EVE artifact, the new steps run as
no-ops:

  1. Confirm eden.yml (the eden-only CI path that consumes
    lfedge/eve:16.6.0) still passes after this change. The
    Collect coverage step warns ("no covcounters appeared")
    because zedbox isn't built with -cover, and the
    Upload coverage artifact step skips because the
    eden_e2e_coverage.txt hashFiles check is false. No new artifact
    is uploaded.
  2. Confirm the coverage-merge fan-in job runs to completion and
    emits the "No raw covdata found across jobs" warning instead of
    failing.
  3. Locally: build EVE with COVER=y and run a single suite via
    make eden-cover (or by passing
    --coverage-dir <dir> to eden test); confirm
    <dir>/eden_e2e_coverage.txt AND <dir>/covdata/coverage/
    (containing covmeta.* and covcounters.*) both exist after
    the run.
  4. Once the companion EVE PR lands on master, exercise the full
    path end-to-end by labelling a coverage-instrumented EVE PR with
    coverage and confirming the covdata-merged artifact is
    produced at the end of the workflow run.

Notes

No user-facing changes. CI / dev-tooling only.

eriknordmark and others added 2 commits May 15, 2026 11:16
CollectEveCoverage previously SCP'd the raw covmeta.* / covcounters.*
files into a temp dir, ran `go tool covdata textfmt` on them, and let
the deferred cleanup wipe the binary files on return. Only the text
profile at <output-dir>/eden_e2e_coverage.txt survived.

Callers that want to merge coverage across multiple runs (e.g. across
the Eden test matrix) need the raw covdata so they can use
`go tool covdata merge`; the text profile alone only supports
concatenation, which is fine for a per-line summary but doesn't
aggregate the way merge does.

Copy the SCP target into <output-dir>/covdata/ instead of an
auto-deleted temp dir. The text-profile output path is unchanged.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each suite in the Eden test matrix runs on its own runner with its own
./eden setup -> start -> stop -> make clean lifecycle, so per-job
collection is the only point at which to capture covdata from a
coverage-instrumented EVE build. After make clean the runner is wiped
and there is no second chance.

Add a Collect/Upload coverage pair to the run-eden-test composite
action: invoke `eden eve collect-coverage` with `if: always()` and
best-effort error handling so a failed test still surfaces whatever
covdata was written, and upload the eden_coverage/ output directory
(both the eden_e2e_coverage.txt text profile and the raw covdata
subtree).

Add a coverage-merge job to test.yml that downloads every covdata-*
artifact, runs `go tool covdata merge` on the raw covmeta.* /
covcounters.* files to produce a single merged covdata directory,
converts it to a text profile via `covdata textfmt`, and runs
`go tool cover -func` for a per-package summary.

Producing useful coverage data requires the EVE artifact consumed by
these jobs to be built with COVER=y so zedbox registers the SIGUSR2
handler and sets GOCOVERDIR=/persist/coverage. That is a separate
change on the EVE side.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@eriknordmark eriknordmark force-pushed the ci-coverage-collection branch from 2744e25 to c900dfb Compare May 15, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant