Test result transfer for multiple matches

[sinanmetin]

Hello Bryan,
I have a question. When there is more than one match for a test case work item with the same test case ID in the test plan, the extension throws an error and does not carry over the test result.

‘##[warning]Multiple matches were found for test case:’ appears.
To prevent this warning, adjust the duplicates or the testCaseMatchStrategy to be more specific.
##[error]Couldn't create a TestRun for this TestPlan because the test results could not be correlated to any known TestCases."

In our use case, the same test case can exist within the same test plan with the same configuration. Is there a way in the extension settings to transfer the test result to all test cases with the same ID in the Test Plan?

Thank you very much for your help.

[bryanbcook]

This came up in #113

In that example, the Test Plan had duplicate Test Cases, e.g add the test suite from a previous Test Plan which resulted in the Test Case being included more than once. The test automation only a produces a single result for that Test Case. The Test Case used the same configuration for both items. Please advise if that is the scenario you're seeing?

In my view, the Test Plan represents the total amount of effort to test, so duplicates are a misrepresentation of that effort.

For an immediate workaround, the suites are malleable when you add them to the Test Plan. Removing the duplicates will prevent the issue.

As this has come up before, I suspect it might occur more, so I'm open to discussion. The current design is 1:1 between test result and Test Point (via TestCase), so switching to 1:many would need some review. At a minimum I can look at improving the error message if the duplicate matches are the same test case.

[bryanbcook]

Also, you are getting a warning about duplicates AND an error that you test automation contains no matches. Can you confirm that your test plan only contains duplicates thus no test results were matched? Or, processing halted because there were duplicates?

[sinanmetin]

I actually used to think that having duplicate tests in a Test Plan doesn’t make much sense either.
However, in the project I’m currently working on, we organize tests in the Test Plan under requirement-based test suites.

Some test cases verify more than one customer requirement, so in a few cases we intentionally reuse the same test case as a duplicate. The idea behind this is twofold:

to clearly see whether all customer requirements are fully covered by realistic test scenarios, and

when a test fails, to easily identify which requirements are impacted.

For that reason, we group all tests that verify a specific requirement under the corresponding requirement-based test suite.

When I create a Test Plan for a release, I usually clean up these duplicates beforehand, and in that case the Test Plan extension transfers the test results without any issues.

I was wondering whether there might be a setting or strategy related to duplicates that would make this cleanup step unnecessary, so that test results could still be transferred even if the same test case appears multiple times in the Test Plan.

Just wanted to share the background from our side and understand whether this is something that could be supported, or if the current 1:1 design is the intended limitation.

[sinanmetin]

Also, you are getting a warning about duplicates AND an error that you test automation contains no matches. Can you confirm that your test plan only contains duplicates thus no test results were matched? Or, processing halted because there were duplicates?

The Test Plan contained around 70 test cases, and only one of them was duplicated.

All other test results were transferred successfully. The issue occurred only for the duplicated test case, whose result was not transferred due to the multiple match warning.

However, because the extension treats this as an error, the pipeline ultimately failed, even though the vast majority of test results were processed correctly.

So processing was not halted upfront because of duplicates — it continued, but failed at the end due to the unmatched duplicate test case.

[bryanbcook]

So processing was not halted upfront because of duplicates — it continued, but failed at the end due to the unmatched duplicate test case.

I'm surprised by this. I would like to see some debug logs showing that it published results and failed because there were duplicates. Because as I see it the Processor only warns when it finds duplicates and the Publisher only throws when there aren't any matches at all

[bryanbcook]

I was wondering whether there might be a setting or strategy related to duplicates that would make this cleanup step unnecessary, so that test results could still be transferred even if the same test case appears multiple times in the Test Plan.

It looks like I should be able to detect when all the duplicate matched test points are for the same test case id. I might log a warning that the test plan has duplicate entries, and I don't think there needs to be a setting to enable or disable that feature.

[sinanmetin]

So processing was not halted upfront because of duplicates — it continued, but failed at the end due to the unmatched duplicate test case.

I'm surprised by this. I would like to see some debug logs showing that it published results and failed because there were duplicates. Because as I see it the Processor only warns when it finds duplicates and the Publisher only throws when there aren't any matches at all

Yes, you were right. I checked it again and it failed another reason

[sinanmetin]

I was wondering whether there might be a setting or strategy related to duplicates that would make this cleanup step unnecessary, so that test results could still be transferred even if the same test case appears multiple times in the Test Plan.

It looks like I should be able to detect when all the duplicate matched test points are for the same test case id. I might log a warning that the test plan has duplicate entries, and I don't think there needs to be a setting to enable or disable that feature.

I’ve just discovered this feature in ADO. If there are multiple instances of the same test case within the same test plan using the same configuration, ADO allows you to automatically carry over the test result to the others. When this option is selected, it is actually sufficient for the test plan extension to carry over the result to a test point of the same test case.

[bryanbcook]

Oh that's interesting. As mentioned above, I have a PR to fix this, just working through some of the other ancillary efforts for updating npm packages, etc. I might need to test what happens if this setting on the test plan is enabled, or if I can detect if that setting has been enabled.

[bryanbcook]

My fixes work against a Test Plan that has the "Show same outcome of the tests in multiple suites under the same plan" setting enabled, though it reports warnings that there are duplicates in the plan. I might need to detect if this setting is enabled and adjust the logging accordingly.