Skip to content

Conversation

@let-def
Copy link

@let-def let-def commented Nov 18, 2025

This PR integrates a fuzzer (ocamlgrammarfuzzer) into the CI. The idea is to derive a deterministic set of sentences from the grammar that guarantees exhaustive coverage of all syntactic constructions, and then to classify the behavior of ocamlformat on these sentences.

HACKING.jst.md has been updated to describe the fuzzer integration.

Catching regressions in the CI

The main job of the CI is to catch regressions, not to guarantee the absence of errors.
Here are sample outputs of the CI catching changes in coverage:

Helping developers improve coverage

For developers, another workflow is provided. Running make fuzz produces reports that give a detailed classification of all failures of ocamlformat, as well as regressions compared to a previous baseline set with make fuzz-update-state.

@let-def
Copy link
Author

let-def commented Nov 18, 2025

I am not sure whether the CI logs can be consulted, so here is the relevant lines for the regression case:

Regression: (* C0 *) exception (* C1 *) false (* C2 *) let (* C3 *) x (* C4 *)
Regression: (* C0 *) val (* C1 *) x (* C2 *) : (* C3 *) ( (* C4 *) {%ext|s|} (* C5 *) ) (* C6 *)
Regression: (* C0 *) kind_abbrev_ (* C1 *) x (* C2 *) = (* C3 *) x (* C4 *) with (* C5 *) {%ext|s|} (* C6 *) -> (* C7 *) {%ext|s|} (* C8 *)
Tested 490799 sentences:
- 251417 successfully formated (51.23%)
- 59851 failed with syntax errors (12.19%)
- 715 had comments dropped (0.15%) (732 comments were dropped in total)
- 178818 caused internal errors (36.43%)
❌ valid sentences: -3 (REGRESSION)
❌ syntax errors: +1 (REGRESSION)
❌ comment errors: +1 (REGRESSION)
❌ comments dropped: +1 (REGRESSION)
❌ internal errors: +1 (REGRESSION)

And in the improvement case:

Tested 490799 sentences:
- 251417 successfully formated (51.23%)
- 59851 failed with syntax errors (12.19%)
- 715 had comments dropped (0.15%) (732 comments were dropped in total)
- 178818 caused internal errors (36.43%)
✅ valid sentences: +3
✅ syntax errors: -1
✅ comment errors: -2
✅ comments dropped: -1
✅ internal errors: -1

Previously, we would just check for the absence of regressions.
Ensuring the fuzzer state is updated prevents a developer from
forgetting to update the baseline when there are no regressions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant