chore: flip filtering label to mark examples with tests #15002 #15014
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gianarb
force-pushed
the
chore/new-example-testing-labeling
branch
7 times, most recently
from
639e09f to
87317dd
Compare
This commit removes the label `workflows.argoproj.io/test` used to mark examples as testable because now we want as contributors to be more explicit marking examples that DO NOT have tests, this should help maintainers to ask for those tests as part of code review. This commit adds `workflows.argoproj.io/no-test-broken` for genuinely broken examples which can be fixed or that do not have test yet. `workflows.argoproj.io/no-test-environment` for examples which need to run in a different environment, e.g. need access to artifactory but this commit does not mark those. Related argoproj#15002 Signed-off-by: Gianluca Arbezzano <[email protected]>
gianarb
force-pushed
the
chore/new-example-testing-labeling
branch
11 times, most recently
from
d2ce12e to
e08c4b6
Compare
…5002 Since we have changed how we identify if an example runs correctly as test this commit applies to the new protocol in the test runner. Signed-off-by: Gianluca Arbezzano <[email protected]>
gianarb
force-pushed
the
chore/new-example-testing-labeling
branch
5 times, most recently
from
36067d9 to
295a2ba
Compare
Many examples could pass tests but they were marked as no-test for various reasons. I analyzed all the tests and I marked the one that did not pass as `no-test-broken` or `-no-test-environment` according to why they were not passing. The most common issues for failing example were the image used as part of the workflow since we have a list of allowed images that can run as tests I expanded the list accordingly and I moved some images that were doing easy tasks for already allowed images. For example we had some debian images that were running `sleep` or `cat`, such commands can run from a busybox and it was already allowed so I moved the workflow to use it. As follow up I will update images to their up to date version. Signed-off-by: Gianluca Arbezzano <[email protected]>
gianarb
force-pushed
the
chore/new-example-testing-labeling
branch
from
295a2ba to
9af4b12
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference: #15002
Motivation
We want to increase the test coverage for our examples since they are important
to guide users and for documentations we want them to run correctly.
Modifications
Thi PR flips how we mark example with validation tests. Previously we were
marking only the examples with a test, it makes this process easy to kind of
chill out and avoid writing a validation test. We decided to run example tests
by default for all examples that do not explicitly exclude. This should help us
to writing those tests.
Verification
As part of exeuction of the tests I wrote the path of the examples that ran as
test in a file prev and after the change to validate that the same number of
examples got executed
Documentation
I could not find documentation mentioning the old label used so I didn't add
it. If you can point me to where I should add it I am happy to write something
about it.