Test that fails to load in a test suite do not count as failed tests

[traversaro]

If the testrunner fails to find the plugin for a given tests, it prints the following erro message:

2: [testrunner] cannot load plugin myTest.so; error (load) : myTest.so: cannot open shared object file: No such file or directory

But the final output is:

2: ---------- results -----------
2: Total number of test suites  : 1
2: Number of passed test suites : 1
2: Number of failed test suites : 0
2: Total number of test cases   : 0
2: Number of passed test cases  : 0
2: Number of failed test cases  : 0

A test that was impossible to load should count as a failed test.

[apaikan]

how we know that the myplugin.so is actually a RTF test plugin when it is failed to load? For example when launching testrunner -r --tests /foo/bar ?

Maybe it make sense only if the plugin is explicitly listed in XML suite file. right?

[traversaro]

Good point @apaikan , title updated!

[apaikan]

Alternative solutions

1. Can we add another parameter to lunch testrunner in restricted mode which count unloaded plugins as failed? such as testrunner --conservative ... or tesrunner --restricted ...
2. Adding another line to the test report as Number of unloaded test cases : 1
3. Considering unloaded tests as failed only if the test case run from a test suite