Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions docs/internals/extensions/rst_filebased_testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ The test files are expected to contain the following format:

#CHECK: <check functions>

#EXPECT: <warning message>
#EXPECT-NOT: <warning message>
#EXPECT[+x]: <warning message>
#EXPECT-NOT[+x]: <warning message>

<need information>

Expand All @@ -36,18 +36,37 @@ Message text which is expected/not expected during the
This message is checked for the Sphinx-Needs directive
specified after the EXPECT/EXPECT-NOT statement.

This message needs a '[+x]'offset after the 'EXPECT/-NOT' that should point to the need
that should (not) emit the warning.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to practically always be [+2]. So it feels more like unnecessary boilerplate.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is currently cause we only have 1 EXPECT there. If we had 2 or more it would be `
EXPECT[+3]: ...
EXPECT[+2]: ...

etc.
It truly is only an offset that EXACTLY tells us where they want the warning to be.
This way we can also safeguard a bit against sphinx not seeing a need but we do.


**\<need information>**<br>
One or more Sphinx-Needs directives needed for the
Sphinx document build

**Example:**

#CHECK: check_options
#EXPECT: std_wp__test__abcd: is missing required attribute: `status`.
#EXPECT[+2]: std_wp__test__abcd: is missing required attribute: `status`.

.. std_wp:: Test requirement
:id: std_wp__test__abcd

This example verifies that the warning message
*std_wp__test__abcd: is missing required attribute: \`status\`*
is shown during the Sphinx build. Only the *check_options* check is enabled.

With the '[+2]' after the 'EXPECT' we tell the parser that we want this warning
to be emitted and checked for 2 lines underneath

There is multiple things that are not allowed for example, you need to have
a new line between the EXPECT/-NOT and the need that it refers to
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There needs to be a new line between the EXPECT/-NOT and the need.
If that is not the case, then our parser sees it correctly (as we just filter for '.. ' and '::') but sphinx-needs or more precisely docutils thinks it is normal text and doesn't actually generate the need itself.
Therefore we have a miss match of what we see and what sphinx sees. This is not fixable on our side (making the need when sphinx doesn't see it). So we have to enforce (lightly) that there is a new line.


**Negative Example**


#EXPECT-NOT[+1]: std_wp__test__abcd: is missing required attribute: `status`.
.. std_wp:: Test requirement
:id: std_wp__test__abcd

This will error and let you know that an offset of '1' is not allowed and you
need to add a new line beneath the Warning Statement
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

.. Test: No external prefixes (single documentation mega-build)
.. Verifies links work when all needs are loaded in one Sphinx instance, without prefix logic.
#EXPECT-NOT tool_req__test_abcd.satisfies (doc_getstrt__req__process): does not follow pattern `^doc_.+$`.
#EXPECT-NOT[+2]: does not follow pattern `^doc_.+$`.

.. tool_req:: This is a test
:id: tool_req__test_abcd
Expand All @@ -27,8 +27,8 @@

.. Also make sure it works with lists of links

#EXPECT-NOT: tool_req__test_aaaa.satisfies (doc_getstrt__req__process): does not follow pattern `^doc_.+$`.
#EXPECT-NOT: tool_req__test_aaaa.satisfies (gd_guidl__req__engineering): does not follow pattern `^gd_.+$`.
#EXPECT-NOT[+3]: does not follow pattern `^doc_.+$`.
#EXPECT-NOT[+2]: does not follow pattern `^gd_.+$`.

.. tool_req:: This is a test
:id: tool_req__test_aaaa
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,31 +14,31 @@
#CHECK: check_id_format

.. Id does not consists of 3 parts
#EXPECT: stkh_req__test.id (stkh_req__test): expected to consist of this format: `<Req Type>__<Abbreviations>__<Architectural Element>`.
#EXPECT[+2]: stkh_req__test.id (stkh_req__test): expected to consist of this format: `<Req Type>__<Abbreviations>__<Architectural Element>`.

.. stkh_req:: This is a test
:id: stkh_req__test

.. Id consists of 3 parts
#EXPECT-NOT: stkh_req__test__abcd.id (stkh_req__test__abcd): expected to consist of this format: `<Req Type>__<Abbreviations>__<Architectural Element>`.
#EXPECT-NOT[+2]: expected to consist of this format

.. stkh_req:: This is a test
:id: stkh_req__test__abcd

.. Id follows pattern
#EXPECT: stkh_req__test__test__abcd.id (stkh_req__test__test__abcd): expected to consist of this format: `<Req Type>__<Abbreviations>__<Architectural Element>`.
#EXPECT[+2]: stkh_req__test__test__abcd.id (stkh_req__test__test__abcd): expected to consist of this format: `<Req Type>__<Abbreviations>__<Architectural Element>`.

.. stkh_req:: This is a test
:id: stkh_req__test__test__abcd

.. Id starts with wp and number of parts is 3
#EXPECT: wp__test__abcd.id (wp__test__abcd): expected to consist of this format: `<Req Type>__<Abbreviations>`.
#EXPECT[+2]: wp__test__abcd.id (wp__test__abcd): expected to consist of this format: `<Req Type>__<Abbreviations>`.

.. workproduct:: This is a test
:id: wp__test__abcd

.. Id is invalid, because it starts with wp and contains 2 parts
#EXPECT-NOT: wp__test.id (wp__test): expected to consist of this format: `<Req Type>__<Abbreviations>`.
#EXPECT-NOT[+2]: expected to consist of this format

.. workproduct:: This is a test
:id: wp__test
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@
#CHECK: check_id_length

.. Id contains too many characters
#EXPECT: std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd.id (std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd): exceeds the maximum allowed length of 45 characters (current length: 47).
#EXPECT[+2]: std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd.id (std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd): exceeds the maximum allowed length of 45 characters (current length: 47).

.. std_wp:: This is a test
:id: std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd

.. Id has correct length
#EXPECT-NOT: exceeds the maximum allowed length of 45 characters
#EXPECT-NOT[+2]: exceeds the maximum

.. std_wp:: This is a test
:id: std_wp__test__abce
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,30 @@


.. Title contains a stop word
#EXPECT: feat_req__test__title_bad: contains a weak word: `must` in option: `title`. Please revise the wording.
#EXPECT[+2]: feat_req__test__title_bad: contains a weak word: `must` in option: `title`. Please revise the wording.

.. feat_req:: This must work
:id: feat_req__test__title_bad



.. Title contains no stop word
#EXPECT-NOT: title
#EXPECT-NOT[+2]: title

.. feat_req:: This is a test
:id: feat_req__test__title_good



.. Title of an architecture element contains a stop word
#EXPECT: stkh_req__test_title_bad: contains a weak word: `must` in option: `title`. Please revise the wording.
#EXPECT[+2]: stkh_req__test_title_bad: contains a weak word: `must` in option: `title`. Please revise the wording.

.. stkh_req:: This must work
:id: stkh_req__test_title_bad



#EXPECT-NOT: title
#EXPECT-NOT[+2]: title

.. stkh_req:: This is a test
:id: stkh_req__test_title_good
Expand All @@ -47,7 +47,7 @@


.. Description contains a weak word
#EXPECT: stkh_req__test__desc_bad: contains a weak word: `really` in option: `content`. Please revise the wording.
#EXPECT[+2]: stkh_req__test__desc_bad: contains a weak word: `really` in option: `content`. Please revise the wording.

.. stkh_req:: This is a test
:id: stkh_req__test__desc_bad
Expand All @@ -57,7 +57,7 @@


.. Description contains no weak word
#EXPECT-NOT: stkh_req__test__desc_good: contains a weak word: `really` in option: `content`. Please revise the wording.
#EXPECT-NOT[+2]: contains a weak word

.. stkh_req:: This is a test
:id: stkh_req__test__desc_good
Expand All @@ -67,15 +67,15 @@


.. Description of architecture view of type feat_arc_sta is not checked for weak words
#EXPECT-NOT: content
#EXPECT-NOT[+2]: content

.. feat_arc_sta:: This is a test
:id: feat_arc_sta_desc_good

This should really work


#EXPECT: tool_req__docs_common_attr_desc_wording: contains a weak word: `just` in option: `content`. Please revise the wording.
#EXPECT[+2]: tool_req__docs_common_attr_desc_wording: contains a weak word: `just` in option: `content`. Please revise the wording.

.. tool_req:: Enforces description wording rules
:id: tool_req__docs_common_attr_desc_wording
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,23 @@
#CHECK: check_validity_consistency


#EXPECT: feat_req__random_id1: inconsistent validity: valid_from (v1.0) >= valid_until (v0.5).
#EXPECT[+2]: feat_req__random_id1: inconsistent validity: valid_from (v1.0) >= valid_until (v0.5).

.. feat_req:: from after until
:id: feat_req__random_id1
:valid_from: v1.0
:valid_until: v0.5


#EXPECT-NOT: feat_req__random_id2: inconsistent validity: valid_from (v0.5) >= valid_until (v1.0).
#EXPECT-NOT[+2]: inconsistent validity

.. feat_req:: until after from
:id: feat_req__random_id2
:valid_from: v0.5
:valid_until: v1.0


#EXPECT: stkh_req__random_id1: inconsistent validity: valid_from (v1.0.1) >= valid_until (v0.5).
#EXPECT[+2]: stkh_req__random_id1: inconsistent validity: valid_from (v1.0.1) >= valid_until (v0.5).

.. stkh_req:: from after until for stakeholder requirement
:id: stkh_req__random_id1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
.. We can not yet enable this test. As the check is only an 'info' and not yet a true warning
.. Therefore the test is the inverse of what we will test once it is enabled.

.. #EXPECT: comp_saf_fmea__child__16: is valid but links to invalid need(s):
#EXPECT-NOT: invalid need(s):

#EXPECT-NOT[+2]: invalid need(s):

.. comp_saf_fmea:: Child requirement
:id: comp_saf_fmea__child__1
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@


.. Positive Test: Child requirement QM. Parent requirement has the correct related safety level. Parent requirement is `QM`.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. feat_req:: Child requirement 1
:id: feat_req__child__1
Expand All @@ -41,7 +41,7 @@


.. Positive Test: Child requirement ASIL B. Parent requirement has the correct related safety level. Parent requirement is `QM`.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. feat_req:: Child requirement 2
:id: feat_req__child__2
Expand All @@ -52,7 +52,7 @@


.. Negative Test: Child requirement QM. Parent requirement is `ASIL_B`. Child cant fulfill the safety level of the parent.
#EXPECT: QM requirements cannot satisfy ASIL requirements.
#EXPECT[+2]: QM requirements cannot satisfy ASIL requirements.

.. comp_req:: Child requirement 3
:id: feat_req__qm_child_with_asil_parent
Expand All @@ -64,7 +64,7 @@

.. Mitigation of Safety Analysis (FMEA and DFA) shall be checked. Mitigation shall have the same or higher safety level than the analysed item.
.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item.
#EXPECT: feat_saf_dfa__child__5: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.
#EXPECT[+2]: feat_saf_dfa__child__5: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.

.. feat_saf_dfa:: Child requirement 5
:id: feat_saf_dfa__child__5
Expand All @@ -74,7 +74,7 @@


.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. feat_saf_dfa:: Child requirement 6
:id: feat_saf_dfa__child__6
Expand All @@ -85,7 +85,7 @@


.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item.
#EXPECT: comp_saf_dfa__child__7: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.
#EXPECT[+2]: comp_saf_dfa__child__7: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.

.. comp_saf_dfa:: Child requirement 7
:id: comp_saf_dfa__child__7
Expand All @@ -95,7 +95,7 @@


.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. comp_saf_dfa:: Child requirement 8
:id: comp_saf_dfa__child__8
Expand All @@ -106,7 +106,7 @@


.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item.
#EXPECT: feat_saf_dfa__child__9: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.
#EXPECT[+2]: feat_saf_dfa__child__9: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.

.. feat_saf_dfa:: Child requirement 9
:id: feat_saf_dfa__child__9
Expand All @@ -116,7 +116,7 @@


.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. feat_saf_dfa:: Child requirement 10
:id: feat_saf_dfa__child__10
Expand All @@ -127,7 +127,7 @@


.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item.
#EXPECT: feat_saf_fmea__child__11: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.
#EXPECT[+2]: feat_saf_fmea__child__11: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.

.. feat_saf_fmea:: Child requirement 11
:id: feat_saf_fmea__child__11
Expand All @@ -137,7 +137,7 @@


.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. feat_saf_fmea:: Child requirement 12
:id: feat_saf_fmea__child__12
Expand All @@ -148,7 +148,7 @@


.. Positive Test: Linked to a mitigation that is higher to the safety level of the analysed item.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. feat_saf_fmea:: Child requirement 13
:id: feat_saf_fmea__child__13
Expand All @@ -158,7 +158,7 @@


.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item.
#EXPECT: comp_saf_fmea__child__14: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.
#EXPECT[+2]: comp_saf_fmea__child__14: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid.

.. comp_saf_fmea:: Child requirement 14
:id: comp_saf_fmea__child__14
Expand All @@ -168,7 +168,8 @@


.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item.
#EXPECT-NOT: safety
#EXPECT-NOT[+2]: safety

.. comp_saf_fmea:: Child requirement 15
:id: comp_saf_fmea__child__15
:safety: ASIL_B
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,23 @@
#CHECK: id_contains_feature

.. Feature is in the path of the RST file
#EXPECT-NOT: Feature 'id_contains_feature' not in path
#EXPECT-NOT[+2]: Feature 'id_contains_feature' not in path

.. std_wp:: This is a test
:id: std_wp__id_contains_feature__abce


.. Check if the feature is in the path of the RST file is skipped,
because the id contains 4 parts
#EXPECT-NOT: not in path
#EXPECT-NOT[+2]: not in path

.. std_wp:: This is a test
:id: std_wp__test1__test2__abce


.. Check if the feature is in the path of the RST file is skipped,
because the requirement type is stkh_req
#EXPECT-NOT: Feature 'test' not in path
#EXPECT-NOT[+2]: Feature 'test' not in path

.. stkh_req:: This is a test
:id: stkh_req__test__abce
Loading
Loading