Ambigous test points should not be ignored

[gruenedd]

Hi,

If more than one test points matches, i would expect that the result is added to each matching test point.
But currently just the warning a raised and nothing happens.
Background is that i have multiple test suites within the test plan and some of them share test cases.

I think this could be changed here in TestResultProcessor.ts

  } else {
    const testCaseNames = matchingPoints.map(item => JSON.stringify(item)).join("\n");
    this.logger.warn(`Multiple matches were found for test case: ${frameworkResult.name}. Matches:\n${testCaseNames}`);
    this.logger.info("To prevent this warning, adjust the duplicates or the testCaseMatchStrategy to be more specific.");

    // ------------------------------------------------------------------------------
    matchingPoints.forEach(p => {
      result.matches.set(p.id, frameworkResult);
    });
    var matchingPointIds = matchingPoints.map(p => p.id);
    testPoints = testPoints.filter(i => matchingPointIds.indexOf(i.id) === -1);
    // ------------------------------------------------------------------------------
  }

Would it be possible to add this?
Or maybe i miss something here and this change isn't even required(?)

Cheers

[bryanbcook]

Depends what you are trying to do.

A TestPoint is different than a TestCase. The TestCase is the definition of steps (for example, logging into a system). But you can have multiple permutations of the identical steps on different systems (Chrome, Safari, Edge, iOS, etc) -- these are Configurations. A TestPoint is a combination of TestPlan+TestCase+TestConfiguration. My understanding is TestPoints are unique within a TestPlan.

I'd be curious to see how you were able to represent the same test case with the same configuration in multiple test suites within a single plan.

Based on this, a TestPoint can only have one outcome (Passed/Failed/Skipped/NotExecuted). If the system allowed more than one test result to associate to a TestPoint and they had different outcomes, which value should be used?

You can represent a TestCase multiple times in your test automation if you also include the TestConfig.

Please let me know if this helps, or if you can demonstrate that the point appears more than once in your test plan.

[bryanbcook]

Also, please include details (YAML, etc) on how you have configured the task. The line of code you've outlined is triggered when the configuration is too lax.

[gruenedd]

Hi, thank for your reply.

Based on this, a TestPoint can only have one outcome (Passed/Failed/Skipped/NotExecuted). If the system allowed more than one test result to associate to a TestPoint and they had different outcomes, which value should be used?

But there are no two different results. It's one test with one result. It just happens that the according test case is within multiple suites within the same test plan.

Cheers

[bryanbcook]

Please provide details (steps to reproduce, screenshots, other evidence) that shows you have two test points with the same id in the same plan. I cannot address nor support you without reproducible evidence.

Please provide the configuration of your task (YAML, screenshots).

[bryanbcook]

I was able to reproduce your setup by importing an existing suite that already had the Test Case into the plan.

But, if you use the following endpoint https://dev.azure.com/<org>/<prj>/_apis/tesplan/Plans/<planId>/<suiteId>/TestPoint?api-version=7.1-preview.2&isRecursive=true, you'll notice that there are separate Test Points (id) for the same testCaseReference. The important thing to remember is when we publish results to the Test Plan, we are publishing outcomes for the Test Points, not the Test Case.

{
  "value": [
        
        {
            "id": 10,
            "testPlan": {
                "id": 168
            },
            "testSuite": {
                "id": 171
            },
            "testCaseReference": {
                "id": 167,
                "name": "Duplicated Test Case",
                "state": "Design"
            }
        },

        {
            "id": 889,
            "testPlan": {
                "id": 168
            },
            "testSuite": {
                "id": 4878
            },
            "testCaseReference": {
                "id": 167,
                "name": "Duplicated Test Case",
                "state": "Design"
            }
        },

  ]
}

I might be able to adjust the logic to determine if the duplicates are for the same testCaseReference, such that the first test framework result would map to the first test point, and then allow for subsequent matches. However, I have the following concerns:

• The order of test framework results is not guaranteed across test frameworks. How would we know looking at a test result that it refers to test point 10 vs 889? If the order of the test results changed between runs, I'd be concerned that the reported results would flip flop between test case instances.
• When using the VSAutomation Property strategy, it is impossible to link a Test Case to multiple automations.

Perhaps I don't understand your motivation to have the exact same test in the same Test Plan and potentially have different results for it. But in my opinion, it is possible to assign different Configurations to each Test Case which would be a better way to disambiguate between your scenarios. e.g.

    [Fact]
    [Trait("TestCaseId", "4831")]
    [Trait("TestConfig", "Windows10")]
    public void Test4831_ForWindows10Scenario()
    {
    }

    [Fact]
    [Trait("TestCaseId", "4831")]
    [Trait("TestConfig", "Windows11")]
    public void Test4831_ForWindows11Scenario()
    {
    }

Let me know if that works for you.

[bryanbcook]

re-reading your response:

But there are no two different results. It's one test with one result. It just happens that the according test case is within multiple suites within the same test plan.

Your Test Plan has ambiguous references to the same Test Case for no effective business reasoning other than you imported it twice. As I've outlined above, the Test Plan represents these as unique Test Points. Reporting of work done in the Test Plan is measured against the number of Test Points. If someone was manually executing these tests, they would need to know that these were duplicates and manually execute one and then replicate the outcome on the duplicate which artificially inflates the reporting of how much work is actually being done. As it stands, the warning identifies that you have duplicated effort in your Test Plan.

Note that when you import a suite into a Test Plan, the system creates a new suite and copies the Test Case references. They are mutable. The correct resolution for your edge case would be to remove the duplicated Test Case from the Test Plan.