Avoid segmentation fault errors during CI tests

[leehart]

Re: issue #835

[leehart]

There appear to be a couple of unexpected linting failures, which I'll also attempt to resolve in this PR.

malariagen_data/util.py:114: error: "Series[Any]" not callable [operator]
malariagen_data/anoph/sample_metadata.py:1158: error: Name "pd.core.series.Series" is not defined [name-defined]

• https://github.com/malariagen/malariagen-data-python/actions/runs/20955557496/job/60219184103?pr=842

[leehart]

Coverage tests failed due to the segmentation fault (pandas/io/parsers/base_parser.py), presumably because the coverage tests still include test_wgs_data_catalog[adir1_sim]. I'll try excluding test_wgs_data_catalog[adir1_sim] from the coverage tests too, to hopefully help confirm where the problem is.

• https://github.com/malariagen/malariagen-data-python/actions/runs/20955557510/job/60219184137?pr=842

[leehart]

Despite excluding the test_wgs_data_catalog[adir1_sim] test in both "tests" and "coverage", observed segmentation fault relating to pandas/io/parsers/base_parser.py, although those tests previously passed in this PR.

• https://github.com/malariagen/malariagen-data-python/actions/runs/20959940220/job/60234083563?pr=842

In that case, the last successful test was:

• tests/anoph/test_sample_metadata.py::test_wgs_data_catalog[ag3_sim] PASSED [ 57%]

This seems to suggest that the segfault might also occur (albeit less frequently) during the test_wgs_data_catalog[af1_sim] case.

I'll try excluding the test_wgs_data_catalog test altogether, to check whether there are any other segfault scenarios.

[leehart]

Noting unexpected test failure, which I guess is random and unrelated:

=========================== short test summary info ============================
FAILED tests/anoph/test_snp_data.py::test_biallelic_snp_calls_and_diplotypes_with_conditions_fractional[ag3_sim] - assert 2 > 2
==== 1 failed, 899 passed, 1 deselected, 340 warnings in 1170.64s (0:19:30) ====

• https://github.com/malariagen/malariagen-data-python/actions/runs/20962327411/job/60242745972?pr=842

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.05%. Comparing base (90da5cc) to head (51f6c4a).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #842      +/-   ##
==========================================
+ Coverage   92.01%   92.05%   +0.03%     
==========================================
  Files          49       49              
  Lines        5175     5175              
==========================================
+ Hits         4762     4764       +2     
+ Misses        413      411       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leehart]

@jonbrenas @tristanpwdennis @ahernank After fixing the linting errors in util.py and anoph/sample_metadata.py, and skipping all the test_wgs_data_catalog tests in both the "tests" and "coverage" CI workflows, all of the CI checks pass.

This seems to confirm that the segfault issue is limited to the test_wgs_data_catalog tests, however we still need to dig deeper to find the exact cause, and then figure out how to fix it.

[leehart]

Noting another unexpected test failure, which I guess is random and unrelated:

=========================== short test summary info ============================
FAILED tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_period_by[ag3_sim-year] - ValueError: No SNPs remaining after applying variant query "effect in ['NON_SYNONYMOUS_CODING', 'START_LOST', 'STOP_LOST', 'STOP_GAINED']".
=== 1 failed, 890 passed, 12 deselected, 309 warnings in 1494.25s (0:24:54) ====

[leehart]

Confusingly, the same tests for the test_wgs_data_catalog sometimes pass for all cases. Perhaps something has recently changed in the background, e.g. data files, or there could be a random factor. I'll try again with the original test function re-enabled, and then I'll try to narrow down the exact cause from there, presuming it will reoccur.

[leehart]

Currently failing to reproduce the segfault failure here, even though it was reproduced a couple of days ago.

I'll try re-running the CI tests on the master branch.

[leehart]

The re-running of code tests on master completed successfully:

• https://github.com/malariagen/malariagen-data-python/actions/runs/20319515792/job/60484952407

But the code tests with code coverage raised the segfault error:

• https://github.com/malariagen/malariagen-data-python/actions/runs/20319515784/job/58371793889

This seems to confirm that this segfault error is rather sporadic.

Also, there was an unexpected error during the latest Notebook tests on the master branch: RequestQuotaExceededError

• https://github.com/malariagen/malariagen-data-python/actions/runs/20319515781/job/58371793964

[leehart]

Segfault reoccured during PR #844

• https://github.com/malariagen/malariagen-data-python/actions/runs/21061927397/job/60570036198?pr=844
• https://github.com/malariagen/malariagen-data-python/actions/runs/21061927397/job/60573174207?pr=844

[leehart]

Noting RequestQuotaExceededError from "notebooks" tests:

• https://github.com/malariagen/malariagen-data-python/actions/runs/21063547376/job/60575346777?pr=842

[leehart]

As an experiment, tried adding engine="python" to every pd.read_csv call in the codebase (around 44 places) but now running into a new set of apparently unrelated ValueError errors, logged under issue #847.

• https://github.com/malariagen/malariagen-data-python/actions/runs/21253793111/job/61162436349