[Reduce] Running grizzly reduce with --no-harness and --strategy list does not record the result #288
Description
When running grizzly.reduce.crash on a crash that includes multiple tests, if --no-harness is evaluated during the initial analysis, no result will be logged. This is because --no-harness will only use the last testcase. This makes it appear as if reduction was successful while not actually doing anything.
➜ test python3 -m grizzly.reduce.crash --xvfb ~/builds/mc-debug/firefox 123456 --strategy list
[2022-03-28 11:48:33] Loaded crash 5456431 (ORIGINAL) from FuzzManager
[2022-03-28 11:48:42] Setting default --tool=grizzly-dominode from CrashEntry
[2022-03-28 11:48:43] Starting Grizzly Reduce
[2022-03-28 11:48:43] Ignoring: log-limit, timeout
[2022-03-28 11:48:43] Running with Xvfb
[2022-03-28 11:48:43] Running for 11 iterations to assess reliability using harness.
[2022-03-28 11:48:43] Checking reliability with only the last testcase.
[2022-03-28 11:48:48] Running test (1/11)...
[2022-03-28 11:48:53] Running test (2/11)...
[2022-03-28 11:48:55] Running test (3/11)...
[2022-03-28 11:48:57] Running test (4/11)...
[2022-03-28 11:49:00] Running test (5/11)...
[2022-03-28 11:49:02] Running test (6/11)...
[2022-03-28 11:49:05] Running test (7/11)...
[2022-03-28 11:49:07] Running test (8/11)...
[2022-03-28 11:49:09] Running test (9/11)...
[2022-03-28 11:49:11] Running test (10/11)...
[2022-03-28 11:49:14] Running test (11/11)...
[2022-03-28 11:49:17] Processing result...
[2022-03-28 11:49:17] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:49:17] Testcase was interesting 9.1% of 11 attempts using harness/last test only.
[2022-03-28 11:49:17] Running for 11 iterations to assess reliability without harness.
[2022-03-28 11:49:17] Only the last testcase of 3 given will be used to assess reliability without harness.
[2022-03-28 11:49:21] Running test (1/11)...
[2022-03-28 11:49:26] Processing result...
[2022-03-28 11:49:26] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:49:30] Running test (2/11)...
[2022-03-28 11:49:36] Processing result...
[2022-03-28 11:49:36] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:49:40] Running test (3/11)...
[2022-03-28 11:49:46] Processing result...
[2022-03-28 11:49:46] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:49:50] Running test (4/11)...
[2022-03-28 11:49:56] Processing result...
[2022-03-28 11:49:57] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:50:02] Running test (5/11)...
[2022-03-28 11:50:08] Processing result...
[2022-03-28 11:50:08] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:50:14] Running test (6/11)...
[2022-03-28 11:50:21] Processing result...
[2022-03-28 11:50:22] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:50:27] Running test (7/11)...
[2022-03-28 11:50:32] Processing result...
[2022-03-28 11:50:32] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:50:37] Running test (8/11)...
[2022-03-28 11:50:42] Processing result...
[2022-03-28 11:50:42] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:50:46] Running test (9/11)...
[2022-03-28 11:50:52] Processing result...
[2022-03-28 11:50:52] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:50:57] Running test (10/11)...
[2022-03-28 11:51:02] Processing result...
[2022-03-28 11:51:02] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:51:06] Running test (11/11)...
[2022-03-28 11:51:12] Processing result...
[2022-03-28 11:51:12] Result: Assertion failure: false (MOZ_ASSERT_UNREACHABLE) (6b635287:71ddcd52)
[2022-03-28 11:51:12] Testcase was interesting 100.0% of 11 attempts without harness/last test only.
[2022-03-28 11:51:12] Last testcase without harness was selected, other 2 testcases in the original will be ignored.
[2022-03-28 11:51:12] Analysis results:
[2022-03-28 11:51:12] * testcase was less reliable with the harness
[2022-03-28 11:51:12] Repeat: 2, Minimum crashes: 2, Relaunch 1
[2022-03-28 11:51:12]
[2022-03-28 11:51:12] Using strategy list (1/1)
[2022-03-28 11:51:12] Testcase list has length 1, not enough to reduce!
[2022-03-28 11:51:12] Reduction summary:
name | duration | successes | attempts | size
----------------------------------------------------------------
init | | | | 4,939,737 (100%)
analysis | 2:29 ( 99%) | 0 ( 0%) | 0 ( 0%) | 110,295 ( 2%)
list | 0s ( 0%) | 0 ( 0%) | 0 ( 0%) | 110,295 ( 2%)
final | 2:29 (100%) | 0 ( 0%) | 0 ( 0%) | 110,295 ( 2%)
Crash ID : 5456431 (grizzly-dominode)
Analysis : last test: 9%, no harness: 100%
Signature : any: False, description: "Assertion failure: false (MOZ_ASSERT_UNREACHABLE)", given: True
Run Parameters : harness: False, min crashes: 2, relaunch: 1, repeat: 2
[2022-03-28 11:51:12] Shutting down...
[2022-03-28 11:51:12] Done.