Fix mutation ordering

[benjeffery]

Addressing #3253

This PR changes the mutation sort order from site->time->ID to site->time->node_time->num_descendants->ID

This is required as if mutations have unknown times, and no parent info, we need to use the node time to have parents before children. For mutations at the same node, we assume the existing order is parents before children. We need to use num descendants to use parent info if it is present for mutations at the same node.

I've updated a few of the breaking Python tests to give a flavour of the impact. This PR still needs tests for the new order, and the rest of the broken tests fixing, but thought this was a good point for feedback.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.74%. Comparing base (d1eb1b4) to head (3104619).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3257      +/-   ##
==========================================
+ Coverage   89.72%   89.74%   +0.01%     
==========================================
  Files          29       29              
  Lines       32500    32456      -44     
  Branches     5949     5940       -9     
==========================================
- Hits        29160    29127      -33     
+ Misses       1894     1888       -6     
+ Partials     1446     1441       -5     

Flag	Coverage Δ
c-tests	86.74% <100.00%> (+0.03%)	⬆️
lwt-tests	80.38% <ø> (ø)
python-c-tests	88.24% <ø> (ø)
python-tests	98.79% <ø> (ø)
python-tests-no-jit	32.62% <ø> (+<0.01%)	⬆️
python-tests-numpy1	50.99% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
c/tskit/tables.c	83.06% <100.00%> (+0.04%)	⬆️
python/tskit/tables.py	99.26% <ø> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jeromekelleher]

What's the difference now between sort_mutations and sort_canonical? Would a simpler way of thinking about this be to say that we're changing things by making sort_mutations simply call sort_canonical? Presumably we thought hard about what sort_canonical had to do, so we're just swapping in that definition now as a stricter ordering. I'd imagine we only kept the original sorting to avoid breakage at the time.

[benjeffery]

The only difference now is that canonical has a sort by node id, which I think is still needed in the case of nodes with equal times?

[jeromekelleher]

Let's just use sort_canonical for now, and see how we get on with test suite breakage? I don't see much point in having another stricter sort if we're already changing the semantics.

[benjeffery]

Sorting is now identical to canonical - only a couple of python tests failing. Will fix those and add some new tests for the new sort order.

[benjeffery]

I've fixed up the tests - the only one I wasn't sure about was tests/test_extend_haplotypes.py::TestExamples::test_suite_examples_defaults[back_mutations] where mutations didn't match between the python and C implementations any more. Digging both methods do a sort during haplotype extension iterations, but the C version does this skipping a mutation sort. Modifying the Python to match fixes the test, but I'm not sure if this has impact on result of extension. Just doing a sort on both at the end didn't fix the error.

[jeromekelleher]

Looks great - I think we need @petrelharp's input on the haplotype extender stuff

[benjeffery]

Whoops - I missed two topology tests. These got a bit more involved - I've had to switch from checking for exact mutational equality to mutational functional equivalence in the simplify tests.

[benjeffery]

I think this is ready for final review. I've added @hyanwong's initial issue failing code as a test.