Skip to content

ci: COVER=y build/test path via coverage label#5950

Draft
eriknordmark wants to merge 1 commit into
lf-edge:masterfrom
eriknordmark:ci-coverage-collection
Draft

ci: COVER=y build/test path via coverage label#5950
eriknordmark wants to merge 1 commit into
lf-edge:masterfrom
eriknordmark:ci-coverage-collection

Conversation

@eriknordmark
Copy link
Copy Markdown
Contributor

Description

Wire end-to-end opt-in coverage collection for Eden runs against EVE.
Adding the coverage PR label causes a parallel COVER=y EVE image
build threaded through PR Gate and the Eden trusted workflow into a
second Eden run, which consumes the instrumented artifact and emits
the covdata-merged artifact via eden's coverage-merge job.

Four pieces:

  • .github/actions/build-eve/action.yml — new cover input
    rebuilds pillar with COVER=y then builds the eve rootfs with the
    instrumented pillar package. The cover artifact is tagged
    evebuild/pr-cover:<pr_id> and uploaded as
    eve-cover-<hv>-<arch>-<platform> so it does not collide with the
    non-cover artifact on the runner or in docker.
  • .github/workflows/build.ymleve-amd64-kvm-generic-cover
    runs in parallel with the existing amd64-kvm-generic build,
    gated on contains(labels, 'coverage'). Non-coverage PRs see this
    job as skipped at workflow start.
  • .github/workflows/pr-gate.yml — Check Build Result fetches
    all jobs from the PR build run once and holds the gate while the
    cover job is in a non-terminal status. The pull_request_review
    trigger fires on review submission regardless of PR build progress,
    so on a coverage-labeled PR the cover build can still be in flight
    at that moment; without the wait, the non-cover build's success
    would let the gate pass and eden-trusted would start
    tests-master-cover before the eve-cover-* artifact was ready.
    Cover-build failure is a soft signal: build_ok is decided by the
    non-cover conclusion, so a failed cover build does not block the
    PR — tests-master-cover will surface the failure loudly by
    failing to download the artifact.
  • .github/workflows/eden-trusted.yml — detect the coverage
    label via gh api, thread it through has_coverage on the context
    job's outputs, and add a tests-master-cover job that runs
    alongside tests-master and consumes the eve-cover-* artifact
    plus the evebuild/pr-cover tag.

The cover path is master-only for now: stable-branch eden tags
(1.0.15) predate eden's eden eve collect-coverage command, so
adding tests-<rel>-cover jobs would have no effect until that eden
change is backported to a new tag.

PR dependencies

Depends on lf-edge/eden#1173 (ci-coverage-collection branch on
eriknordmark/eden):

  • collect-coverage: preserve raw covdata in output — exposes the
    raw covmeta.* / covcounters.* files in <output-dir>/covdata/
    alongside the existing text profile, so go tool covdata merge can
    be used across runs.
  • ci: collect eden e2e coverage per matrix job — adds a
    best-effort Collect coverage / Upload coverage artifact pair
    to run-eden-test, and a coverage-merge fan-in job to
    test.yml that produces the covdata-merged artifact this PR
    expects at the end of each Eden run.

This PR's tests-master-cover job calls into
lf-edge/eden/.github/workflows/test.yml@master, so the eden change
must be merged first; otherwise the cover job will succeed at running
tests but emit no covdata artifact.

How to test and validate this PR

  1. Land the eden dependency PR first.
  2. Open a draft PR against lf-edge/eve with the coverage label
    set, against master.
  3. Confirm in the PR build workflow that both
    eve (amd64, kvm, generic) and eve (amd64, kvm, generic, cover)
    jobs run and upload eve-kvm-amd64-generic and
    eve-cover-kvm-amd64-generic artifacts respectively.
  4. Confirm in the PR Gate workflow that the gate passes only
    after both build jobs have reached a terminal state, even if a
    review is submitted while the cover build is still in flight.
  5. Confirm in the Run Eden workflow that both tests-master and
    tests-master-cover run, and that the cover variant's
    coverage-merge job produces a covdata-merged artifact
    containing combined_coverage.txt and coverage-summary.txt.
  6. Repeat (2)–(5) on a PR without the coverage label and
    confirm:
    • The eve-amd64-kvm-generic-cover job is skipped at workflow
      start (no extra build cost).
    • PR Gate's Check Build Result reads cover_status == "completed"
      (skipped is a terminal state) and falls through to the existing
      non-cover gate logic with no behavior change.
    • tests-master-cover is gated off by has_coverage == 'false'
      and does not run.

Changelog notes

No user-facing changes. Internal CI only.

PR Backports

  • 16.0-stable: No — depends on an eden change that is not in the
    eden tag pinned by eden-trusted.yml for 16.0-stable.
  • 14.5-stable: No — same reason.
  • 13.4-stable: No — same reason.

Coverage on stable branches can be enabled later by backporting the
eden change to a new 1.0.x tag and then mirroring this PR with
tests-<rel>-cover jobs pinned at the new tag.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device — not applicable (amd64 only
    for the initial cover matrix entry)
  • I've written the test verification instructions
  • I've set the proper labels to this PR

Wire end-to-end opt-in coverage collection for Eden runs against
EVE. Adding the `coverage` PR label causes a parallel COVER=y EVE
image build threaded through PR Gate and the Eden trusted workflow
into a second Eden run, which consumes the instrumented artifact and
emits the covdata-merged artifact via eden's coverage-merge job.

build-eve composite action: a new `cover` input rebuilds pillar with
COVER=y then builds the eve rootfs with the instrumented pillar
package. The cover artifact is tagged `evebuild/pr-cover:<pr_id>` and
uploaded as `eve-cover-<hv>-<arch>-<platform>` so it does not collide
with the non-cover artifact on the runner or in docker.

build.yml: `eve-amd64-kvm-generic-cover` runs in parallel with the
existing amd64-kvm-generic build, gated on
`contains(labels, 'coverage')`. Non-coverage PRs see this job as
skipped at workflow start.

pr-gate.yml: Check Build Result fetches all jobs from the PR build
run once and holds the gate while the cover job is in a non-terminal
status. The pull_request_review trigger fires on review submission
regardless of PR build progress, so on a coverage-labeled PR the
cover build can still be in flight at that moment; without the wait,
the non-cover build's success would let the gate pass and
eden-trusted would start tests-master-cover before the eve-cover-*
artifact was ready. Cover-build failure is a soft signal: build_ok
is decided by the non-cover conclusion, so a failed cover build does
not block the PR -- tests-master-cover will surface the failure
loudly by failing to download the artifact.

eden-trusted.yml: detect the `coverage` label via `gh api` (JSON mode
bypasses the classic-Projects GraphQL deprecation that breaks
`gh pr view` on lf-edge repos), thread it through `has_coverage` on
the context job's outputs, and add a `tests-master-cover` job that
runs alongside `tests-master`. The cover job consumes the eve-cover-*
artifact and the evebuild/pr-cover tag.

The cover path is master-only for now: stable-branch eden tags
(1.0.15) predate eden's `eden eve collect-coverage` command, so
adding tests-<rel>-cover jobs would have no effect until that eden
change is backported to a new tag.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant