Add a dummy site at the end of the sequence

[Duncan-JR]

Since we use half-open intervals in tskit, and tsinfer works with discrete sites, ancestors that include the rightmost inference site (let's call its index r) have to be defined as ending at site index r+1. Since this extra site has no position or haplotypes, we have to handle it as a special case at times, hard-coding it to have a position of ts.sequence_length.

This work-in-progress PR seeks to explicitly encode this rightmost extra site, which I've tentatively called a breakpoint, in the ancestor data. This is the first step toward adding support for multiple user-provided breakpoints; for example, the user could specify the start of the centromere as a breakpoint, such that tsinfer would run on each chromosome arm separately.

So far, I've implemented the extra site in the Python engine up until ancestor generation. Next step is to get ancestor and sample matching to work. Here is a small simulated example to show how the change affects ancestral haplotypes:

[jeromekelleher]

I'm not sure "breakpoint" here is a great bit of terminology as @hyanwong points out. It can be confused with ts.breakpoints() which is quite a different thing.

How about terminal_positions as the name of the array in AncestorData? (This is basically what you suggested Yan?)

[codecov]

Codecov Report

❌ Patch coverage is 97.74436% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.35%. Comparing base (dc026b7) to head (5b38506).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1058      +/-   ##
==========================================
+ Coverage   90.26%   90.35%   +0.09%     
==========================================
  Files          19       19              
  Lines        7086     7205     +119     
  Branches     1170     1187      +17     
==========================================
+ Hits         6396     6510     +114     
- Misses        559      562       +3     
- Partials      131      133       +2     

Flag	Coverage Δ
C	81.57% <100.00%> (+0.16%)	⬆️
c-python	58.86% <85.71%> (+0.26%)	⬆️
python-tests	95.68% <99.04%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Python API	95.80% <99.04%> (+0.01%)	⬆️
Python C interface	66.97% <81.81%> (+0.18%)	⬆️
C library	88.68% <100.00%> (+0.09%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Duncan-JR]

I finally got all the tests working (just need to fix some CI issues with Python 3.13). The main issue was that many tests create custom ancestor data/ tree sequences without a terminal site. I first tried adding a site in before matching, but I realised that this doesn't make sense: both match_ancestors and match_samples behave the same way whether a terminal site is present or not. To prevent breakage, after sample matching, ARG edges that end at the terminal position are now extended to sequence length by default. This can also be done after ancestor matching, but this is only necessary in a few tests.

I still need to clean up the edge extension code and add some tests, so it isn't ready for review quite yet.

[jeromekelleher]

Overall looks good. Some low-level changes would help. I can take a look when these are done,.

[Duncan-JR]

@jeromekelleher this is ready for another review. Changing the low-level code to have a separate terminal site function simplified things considerably. Both match_ancestors and match_samples now always carry through any terminal site to the output and edges are unmodified until post-processing. This meant that I had to manually fix every test that didn't post-process the output tree sequence, but I think this is the most elegant solution by far.

[jeromekelleher]

Great, thanks @Duncan-JR. This looks good to me - I'll pick it up next week once I've cleared off a couple off a couple of things. I'll take it from here. Can you squash the commits down please?