Testing

[delatrie]

Context

This PR sets up a testing infrastructure for allure-csharp packages. It also includes a set of Allure.NUnit tests to verify the infrastructure.

The testing scheme

Here is the testing scheme that includes the testing infrastructure, a test project a target project/package, a set of external dependencies, and temporary artefacts generated by the infrastructure.

Sample projects are fully decoupled from tests. They have their own dependency, build configuration, target framework, etc.

The flow

There are two types of the flow: on-demand and pre-run, each consisting of four phases. Each phase can be run manually. Some are run automatically.

Phase	Description	When run automatically	How to run manually
Generate	During this phase the sample solution is created or updated. If it already exists, only modified files are updates to support incremental builds.	On test project build.	dotnet msbuild -t:Allure_GenerateTestSamples
Build	In this phase, the sample project or entire solution is built against a selected configuration and framework.	Right after "Generate," if Allure_PreBuildTestSamples is true. Right before "Run" requested by a test if hasn't been built yet.	dotnet msbuild -t:Allure_BuildTestSamples
Run	In this phase, the sample project is run. The result files are generated in the configured directory.	When requested by a test in on-demand mode.	dotnet msbuild -t:Allure_RunTestSamples
Assert	In this phase, the generated Allure results are validated by the tests.	When a test is run.	dotnet run --project <Testing Project>

On-demand flow

In on-demand flow, the sample projects run when explicitly demanded by a test. It may slightly vary, but typically looks like this:

1. A developer runs dotnet run --project Allure.NUnit.Tests.
2. A build is performed. The sample projects are updated if necessary during the build.
3. A test starts and asks the runner to execute the sample. The runner invokes dotnet test against the sample project.
4. The sample project is restored and built by dotnet test.
5. The tests in the sample projects are run.
6. The runner collects the result files and returns them to the test.
7. The test asserts the values.

This flow allows to run a single test more quickly. On the other hand, running a bunch of tests may be way too slow as each requires to spin a new dotnet test process. Additionally, the tests can't run in parallel because deparate dotnet test processed may conflict with each other when trying simultaneously rebuild common dependencies.

For such scenarios, pre-run flow was designed.

Pre-run flow

Pre-run flow implies running all the samples in advance. The tests are only asserting the already prepared results. This can be performed during the development but the primary goal of this flow is to speed up the CI pipeline. The flow looks like this:

1. A developer runs dotnet run --project Allure.NUnit.Tests -p:Allure_PreRunTestSamples=true
2. A build is performed. The sample solution is updated, restored, built, and tested during the build.
3. A test starts and asks the runner to execute the sample. The runner skips dotnet test. Instead, it reads the default results directory. If it exists and contains result files, the results are returned to the test. Otherwise, the runner throws an error.
4. The test asserts the values.

In a CI pipeline, a more expressed version can be used to test against the default configuration and .NET version:

1. dotnet restore
2. dotnet build --no-restore -p:Allure_PreRunTestingFlow=true -p:Allure_PreBuildTestSamples=true
3. dotnet msbuild -t:Allure_RunTestSamples
4. dotnet run --project Allure.NUnit.Tests --no-restore --no-build

Since no extra dotnet processes are spinned, all tests in all test projects can run in parallel and the run is much faster in general.

MSBuild properties and items

The following items need to be defined for each testing project:

• SamplePackageReference: each item will be converted to a PackageVersion item in the sample solution's Directory.Packages.props. Additionally, each item will be converted to a PackageReference for every sample project, unless Optional is set to true.
  Supported metadata:
  • Version: a version of the package to refer
  • Optional: if set, the package will only be added to Directory.Packages.props. It won't be added to a sample project's package references.
• ProjectUnderTest: each item will be converted to a ProjectReference of a PackageReference for all sample projects, depending on the value of the Allure_TestTargetVersion property:
  • If Allure_TestTargetVersion is unset, ProjectUnderTest becomes ProjectReference.
  • Otherwise, ProjectUnderTest becomes PackageReference with Version set to Allure_TestTargetVersion. A special value SNAPSHOT indicates the current version ($(Version)).
    The motivation is to allow testing against a packed version of the target project (including already published versions).

The following properties can be used to tune the testing process:

• Allure_SampleSupportedTargetFrameworks: the list of target frameworks the sample projects can be compiled against. By default, the list includes one current TFM (as of today, it's net8.0). The motivation is to allow testing agaisnt multiple versions of .NET without creating extra jobs or while being on a developer's machine.
  This property only affects the generation phase.
• Allure_PreBuildTestSamples: if set to true, the sample solution will be restored and built right after the generation.
  This property only affects the generation phase.
• Allure_PreRunTestSamples: if set to true, the sample solution will be restored, built, and tested right after the generation. Additionally, it sets Allure_PreRunTestingFlow to true unless it's defined at the CLI level or redefined at project level.
  This property only affects the generation phase.
• Allure_PreRunTestingFlow: if set to true, enables the pre-run flow. This property affects the generation and execution phases.
• Allure_SampleSelectedTargetFramework: a specific TFM to run the samples against. If not set, the samples are run against each TFM in the Allure_SampleSupportedTargetFrameworks list.
  This property only affects the execution phase.
• Allure_SampleConfiguration: a specific configuration used to build the sample projects.
  This property only affects the build phase.
• Allure_TestTargetVersion: if set, the sample projects reference the target as a NuGet package of the specified version. A special value SNAPSHOT indicated the current development version. In such a case, the package must exist in the local repository pointed by Allure_LocalNugetRepository before the sample solution is built. If unset, the sample projects reference the target via ProjectReference items.
  One side effect is that ProjectReference causes all transitive dependencies to be built on dotnet test (unless skipped by incremental build). Since such dependencies (like Allure.Net.Commons) are shared by multiple sample projects, parallel execution of such dotnet test commands leds to conflicts on the file system level. That's why parallel test execution is disabled if a ProjectReference item is detected in the sample projects.

The following properties affects the testing but have default values and it's unlikely someone needs to change them:

• Allure_LocalNugetRepository: a path to the local NuGet repository that contains unpublished packages (like SNAPSHOT packages).
• Allure_PackageCacheDirectory: a pack to the package cache directory. By default it's ./packages/. The motivation is to ensure SNAPSHOT packages don't pollute the system-wide cache.
• Allure_SampleSolutionName: a name of the sample solution. By default, it's the name of the test project, suffixed with .Samples.
• Allure_SampleSolutionDir: a directory of the sample solution. By default, it's ./artifacts/samples/<test project>.
• Allure_SampleResultsDirectoryFormat: a format string that has one {0} placeholder for the sample ID. When the ID is inserted, it must give the absolute path to a directory. This directory is where the sample runner will look for Allure result files in pre-run flow.

Closes #362.