Support ES&S Multi-Contest CVRs

[artoonie]

Closes #981

[artoonie]

This PR found a bug:

While Apple Numbers only shows non-empty rows, the underlying XLSX XML data can store empty rows. That led to some CVRs having hidden empty rows which we erroneously marked as undervotes. We now correctly detect these rows as rows to be completely ignored and thrown out.

[artoonie]

TODO: Audit log should say "read X ballots, of which Y were ignored" or similar, rather than X=only non-empty ballots

[artoonie]

TODO: Audit log should say "read X ballots, of which Y were ignored" or similar, rather than X=only non-empty ballots

Rather than modifying that row, I opted to add a warning so (A) it's more visible, and (B) we don't need to propagate what is likely source-specific values up from the ES&S reader through the generic reader and into the tabulator session.

It says "Ignored %d rows with no votes for any candidates"

[yezr]

Testing I found that setting Treat Blanks As Undeclared Write-In to true has some unexpected behavior now. When it sees a fully blank record it calls handleEmptyCells at the beginning of endCvr(). It writes into each of those blank ranks an undeclared write-in ranking.

However, the hasSeenAnyNonBlankCandidateCells boolean is only updated in the cell data call back, which never gets fired because all cells are blank. The ranking of all Undeclared Write-In ranks is then filtered out.

Maybe we need to set the hasSeenAnyNonBlankCandidateCells to true when we set a UWI?

[artoonie]

It seems that Treat Blanks As Undeclared Write-In is fundamentally incompatible with this approach. Either blanks mean UWI or it means Unrelated Contest. I wonder if we should go back to the drawing board and consider looking at the other columns to disambiguate. Or, better yet -- find out what's actually possible in the ES&S data format so we can support it without guessing.

If we want to keep this approach, I suggest changing the Treat Blanks As Undeclared Write-In from a checkbox to a dropdown: Blanks are... > [Ignored, Treated as Undeclared Write-Ins]

[yezr]

If we want to keep this approach, I suggest changing the Treat Blanks As Undeclared Write-In from a checkbox to a dropdown: Blanks are... > [Ignored, Treated as Undeclared Write-Ins]

I think you are right that it needs to be mutually exclusive. For our options

• default option of Any Blank Invalid - Causes Tabulation to Fail
• for any single blank being treated as a UWI, the current behavior when the Treat Blanks As Undeclared Write-In is checked, it now says Any Blank Valid - Individual Blanks Treated As UWI
• I think for the Ignored option we might change the description to make it explicit that it is only ignored if it is a full ranking of blanks. Maybe Valid Only As An All Blank Ranking Ballot - Not included in tabulation (For Multi-Contest Exports)

We can work on this vocab...

[artoonie]

Makes sense.

Suggested vocab:

1. Invalid (fail tabulation)
2. Undeclared Write-In (valid if only one blank per voter)
3. Irrelevant Contest (skip row if all rankings are blank)

With 3, the CSV field listing the number of voters who didn't vote at all (Undervotes-No Ranking) would always be zero, and that's invalid. Does it make sense to write "unknown" instead of zero for that field?

[artoonie]

This change is getting larger than it seemed as first, so we'll definitely need to add tests to handle each of these options. The code isn't ready yet, but the functionality is more or less there if you want to give it an initial test @yezr to verify this is the right direction.

[yezr]

As we got into this ticket, we started to see clearly that getting some of our assumptions confirmed about the structure of the ES&S CVR export format would help. I was able to speak to ES&S reps about the CVR export and confirmed some assumptions.

• Most importantly, I confirmed that it is always the text "undervote" when a ballot has an option to rank/vote in a contest and no ranking/vote is made.
• When a ballot is included in an CVR export and a contest is included in that export that doesn't appear on that ballot, the rankings/votes for that contest will all be blank.
• For write-ins, there is an option in the export to either 1) insert a picture of the write-in portion of the ballot in the cell of that ranking. That picture is interpreted in our parser as a blank OR 2) the text "Write-in". I'm going to find a CVR export that has the pictures in it to confirm that. If we could at least differentiate between a picture and a blank that might help us even more.

That should at least remove the need to handle blanks as skipped ranks.