Allocated memory is wrong when running multiple benchmarks

[AdrianoAE]

I have tested this with 2 different setups, 1 big benchmark class with all the benchmarks inside grouped by category and a 2nd scenario where each group is it's own benchmark and then I run using multiple BenchmarkConverter.TypeToBenchmarks with the config

IConfig config = ManualConfig.Create(DefaultConfig.Instance)
	.WithOptions(ConfigOptions.JoinSummary)
	.WithOptions(ConfigOptions.DisableLogFile);

BenchmarkRunner.Run
(
	[
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.Empty), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.Full), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.NumericOnly1), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.NumericOnly9), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.NumericSingleString), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.NumericThreeStrings), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.NumericTwoStrings), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.SingleString), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.StringsOnly10), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.StringsOnly4), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.ThreeStrings), config),
		BenchmarkConverter.TypeToBenchmarks(typeof(ModelAllocationBenchmark.TwoStrings), config)
	]
);

Example benchmarkclass

public static partial class ModelAllocationBenchmark
{
	[GcForce]
	[MemoryDiagnoser]
	[CategoriesColumn]
	[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
	[HideColumns("Job", "StdDev", "RatioSD")]
	public class Empty
	{
		[Benchmark(Baseline = true, Description = Description.C)]
		[BenchmarkCategory(Category.Empty)]
		public CModel C_Empty()
			=> new()
			{
				// ...
			};

		[Benchmark(Description = Description.T)]
		[BenchmarkCategory(Category.Empty)]
		public TModel T_Empty()
			=> new()
			{
				// ...
			};

		[Benchmark(Description = Description.F)]
		[BenchmarkCategory(Category.Empty)]
		public FModel F_Empty()
			=> new()
			{
				// ...
			};
	}
}

Also tested running with .Net 9 and 10 preview 7, the end result is always inconsistent across runs and the allocated memory errors also change across runs

[timcassell]

What version of BenchmarkDotNet are you using? What are the expected results? Can you share a repro project?

[AdrianoAE]

Version 0.15.2
The expected result is that the Allocated memory is consistent across runs, the benchmark themselves are very simple, hardcode creation of the class instance

For example,. just did another run, the values are consistent when they show up

I can try to create a repro, unfortunately cannot share this project

The issue is also present in 0.15.1

[AdrianoAE]

Repro project, there are some runs where it does not happen, but it does happen for most runs for me

[timcassell]

We changed how we measure allocation in v0.15.2, however I am able to repro in both v0.15.2 and v0.15.1. @Maoni0 Any idea why this happens?

[timcassell]

It repros in .Net 9 and 10, doesn't repro in .Net 8. I reported to runtime. dotnet/runtime#118826