From 67d2f1b861cf7ff3c7974df4e42fd1bb3085cb70 Mon Sep 17 00:00:00 2001 From: Chris Langhans Date: Mon, 11 May 2026 11:02:12 +0200 Subject: [PATCH] docs(decisions): add DR-001 for unit test design Documents the four design decisions made in PR #94: - py_itf_unittest macro (thin wrapper, no plugin machinery) - surgical Bazel target splitting for atomic deps - pytest bootstrap via shared main.py - pytest-mock over unittest.mock References: eclipse-score/itf#94 --- docs/decisions/DR-001-infra.md | 136 +++++++++++++++++++++++++++++++++ docs/decisions/index.rst | 33 ++++++++ docs/index.rst | 1 + 3 files changed, 170 insertions(+) create mode 100644 docs/decisions/DR-001-infra.md create mode 100644 docs/decisions/index.rst diff --git a/docs/decisions/DR-001-infra.md b/docs/decisions/DR-001-infra.md new file mode 100644 index 0000000..b4fc3b3 --- /dev/null +++ b/docs/decisions/DR-001-infra.md @@ -0,0 +1,136 @@ + + +# DR-001-Infra: Unit Test Infrastructure Design + +**Date:** 2026-05-11 +**Status:** Accepted +**PR:** [eclipse-score/itf#94](https://github.com/eclipse-score/itf/pull/94) +**Discussion:** [eclipse-score/discussions#2867](https://github.com/orgs/eclipse-score/discussions/2867) + +> This record follows the Decision Record convention established by the +> Eclipse S-CORE project: +> [eclipse-score/score — docs/design_decisions](https://github.com/eclipse-score/score/tree/main/docs/design_decisions). + +## Overview + +This decision record documents the infrastructure design for unit testing in +ITF. It covers the Bazel macro, dependency scoping strategy, pytest bootstrap +pattern, and mocking library choice, all accepted as part of PR #94. + +## Problem Statement + +ITF previously had only integration tests: tests that start a real target +(Docker or QEMU) and exercise the system end-to-end. Adding unit tests raised +four concrete questions that each had multiple viable answers: + +1. Should unit tests reuse `py_itf_test` or have a dedicated macro? +2. How should Bazel dependencies be scoped to keep tests atomic? +3. How does pytest run inside Bazel, and what does that mean for test + structure? +4. Which mocking library should be used? + +## Options Evaluated + +### Macro design + +**Option A — Reuse `py_itf_test` with empty `plugins`.** +The macro would not crash with an empty plugin list, but it would still +generate the launcher script and resolve `PyItfPluginInfo` providers. The +BUILD file would not communicate that no target is involved. + +**Option B — Dedicated `py_itf_unittest` macro (chosen).** +A thin wrapper around `py_test` with no plugin machinery. The name makes +intent explicit. `pytest-mock` is included as a default dep. JUnit XML +reporting is baked in via `$XML_OUTPUT_FILE`. + +### Dependency scoping + +**Option A — One large Bazel target per package.** +Simple to maintain, but pulls in all transitive dependencies as runfiles. +Bazel measures coverage over all files in the runfiles tree, so the coverage +denominator grows with every transitive dep, even ones not under test. + +**Option B — Surgical target splitting (chosen).** +Split Bazel targets along cohesion boundaries so each unit test can declare +only the module it actually exercises. Example: `score/itf/plugins/qemu/BUILD` +was split into `:config` (Pydantic schema only) and `:qemu` (full plugin). The +unit test for schema validation depends only on `:config`, excluding process +management, SSH, and QEMU binary wrappers from its runfiles tree. + +### Pytest bootstrap + +**Option A — `score_py_pytest` from `@score_tooling`.** +The tooling repository provides a `score_py_pytest` rule, but it bundles a +full Python development environment including `basedpyright` and +`nodejs-wheel-binaries`. These are unrelated to the code under test and expand +the runfiles tree significantly, inflating the coverage denominator and +increasing build time. + +**Option B — Shared `main.py` entry point (chosen).** +`py_test` requires an executable Python module. A minimal `main.py` that calls +`pytest.main(sys.argv[1:])` is the de facto standard for Bazel + pytest. The +same bootstrap file is shared across integration and unit test rules, keeping +the approach consistent. This was confirmed as the community standard in the +GitHub discussion linked above. + +### Mocking library + +**Option A — `unittest.mock.patch` via context managers.** +Part of the standard library, no extra dep. Context manager nesting becomes +verbose when multiple objects need patching. + +**Option B — `pytest-mock` via the `mocker` fixture (chosen).** +Patches are registered and torn down automatically through the pytest fixture +lifecycle, removing context manager nesting. Cleaner for tests that mock +several collaborators: + +```python +def test_ping_reachable(mocker): + mocker.patch("score.itf.core.com.ping.shutil.which", return_value="/usr/bin/ping") + mocker.patch("score.itf.core.com.ping.os.system", return_value=0) + assert ping("127.0.0.1") is True +``` + +## Decision & Rationale + +All four decisions favour the option that minimises coupling and maximises +clarity in the BUILD file: + +- **Dedicated `py_itf_unittest` macro** — the name signals "no target" and + the macro carries no plugin machinery. +- **Surgical Bazel target splitting** — dep declarations in BUILD files become + a lightweight design signal: a test that can only list `:config` as a dep + proves that the schema module is cohesive and has no hidden coupling. +- **Shared `main.py` bootstrap** — consistent with integration tests and + aligned with community practice. +- **`pytest-mock`** — included as a default dep in `py_itf_unittest`; test + authors get `mocker` without an explicit declaration. + +Coverage uses Bazel-native LCOV (`configure_coverage_tool = True` in +`MODULE.bazel`) rather than `pytest-cov`, for consistency across all test +types and compatibility with Bazel's `--combined_report`. + +## Key Implications + +- Unit tests live in `test/unit/` and integration tests in `test/integration/`. + The split is enforced by directory layout and BUILD files, not just naming. +- Adding unit tests for a new module may require splitting its Bazel target if + the current target has a large transitive dep set. This is intentional: + splitting is a design signal that the module has a cohesion opportunity. +- `py_itf_unittest` does not support the `plugins` attribute. A test that + needs a real target belongs in `test/integration/` and uses `py_itf_test`. +- The `mocker` fixture preference applies project-wide; `unittest.mock` context + managers should not be introduced in new tests. diff --git a/docs/decisions/index.rst b/docs/decisions/index.rst new file mode 100644 index 0000000..c21b3f1 --- /dev/null +++ b/docs/decisions/index.rst @@ -0,0 +1,33 @@ +.. + # ******************************************************************************* + # Copyright (c) 2026 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. _itf_decisions: + +Decisions +========= + +Design decisions and their rationale, following the convention established by +the `Eclipse S-CORE project `_. + +Each record is named ``DR-{number}-{category}.md`` where category is one of +``arch``, ``infra``, ``proc``, or ``strat``. + +Infrastructure +-------------- + +.. toctree:: + :maxdepth: 1 + :glob: + + DR-*-infra* diff --git a/docs/index.rst b/docs/index.rst index ec37ae0..f26fe47 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -49,6 +49,7 @@ Integration Test Framework for ECU testing in automotive domains. how-to/index reference/index concepts/index + decisions/index manual/index release/index safety_mgt/index