MAINT:update boost to fix skewnorm.ppf

[dschmitz89]

Reference issue

Supersedes #20125
Closes #20124

What does this implement/fix?

The skewnorm quantile calculation was fixed upstream, so this updates boost to the latest commit of the development branch.

Additional information

CC @maresb in case you have more stress tests for the skewnorm distribution.

I added the backport-candidate label as this is a bug fix but feel free to ignore @tylerjereddy as boost had many moving parts lately. Might be worth to run wheel tests as boost was prone to subtle bugs, will leave the decision to people more involved in that though.

[maresb]

Thanks a lot @dschmitz89 for following up on this! I commented in boostorg/math#1120 (comment), but I'm hopeful that your work here will be helpful for my current applications.

The way scipy patches Boost's skewnorm CDF here is really helpful. But unfortunately if we want to use Boost's PPF solver, that will use Boost's unstable CDF function, so it will be plagued by the upstream inaccuracies.

from scipy.stats import skewnorm
from scipy.stats._boost import _skewnorm_cdf
from scipy import __version__

__version__  # '1.10.0'
skewnorm.cdf(-3, 5)  # 4.136670779725204e-55
_skewnorm_cdf(-3, 0, 1, 5)  # 2.3852447794681098e-18

With Mathematica, N[CDF[SkewNormalDistribution[0, 1, 5], -3], 20] gives 4.1366707600635155404e-55, so we see that scipy's numerical integration is very close, while Boost is way off.

I think I know how to stably compute the CDF without numerical integration, but I'm not sure. I'd need to first implement it and test it thoroughly. I'd like to contribute it upstream to Boost and get rid of the numerical integration fallback here, but I'm not sure when I'll be able to get around to it.

[dschmitz89]

Fixing boost's CDF would be very helpful, thanks for looking into that still. Do you agree that the update for the PPF here is an improvement already?

[maresb]

I don't have the time to investigate in-depth, but you seem to have fixed my original test case to fairly good precision. Barring any regressions, which hopefully the test suite mostly prevents, this appears to me like a substantial improvement! 🎉

Thanks again @dschmitz89!

[maresb]

For my current application I only need to determine the PPF of roughly the first percentile. Boost's CDF should be safe as long as the quantile can be expressed as the difference between two floats of order 1. I think that's what this achieves. It'd be really excellent if this works! 🤞

[dschmitz89]

I went ahead and tested the wheels: all pass, one test failure is unrelated.

[mdhaber]

Is it the change to Boost that made the wheel builder tests happen? NM. I see the commment ; )

[mdhaber]

@dschmitz89 This update seems to fix gh-19348 and gh-18906 on my end, too (or they were already fixed). Could you check on your side, and if so add a test?

Update - seems like they were already fixed by something? I'm closing those issues. If you can confirm locally that they appear to be resolved by running the tests mentioned in my last posts (in each issue), I'd appreciate it!

[dschmitz89]

@dschmitz89 This update seems to fix gh-19348 and gh-18906 on my end, too (or they were already fixed). Could you check on your side, and if so add a test?

Update - seems like they were already fixed by something? I'm closing those issues. If you can confirm locally that they appear to be resolved by running the tests mentioned in my last posts (in each issue), I'd appreciate it!

Good point, I had already forgotten about those. I can confirm that both issues are resolved in main, probably due to the Boost update in #20393 .

[mdhaber]

My only hesitation with merging here is the submodule update, since people tend to find it disruptive and it was already updated quite recently. Would you like this to merge now, or can it wait until release time (at which point we might update to latest Boost)?

[dschmitz89]

I agree. For the sake of not annoying contributors shortly before a release, let's delay until after the 1.14 release.

[mdhaber]

Oh, actually, I still wanted this to make it into 1.14. Let's just ask others who might have an opinion:

@tylerjereddy @rgommers this update to the Boost submodule would close gh-20124. There was just an update to the Boost submodule in gh-20393, and I know we don't like submodule updates to be too frequent. When would be the best time to merge this - now, wait a bit but merge before 1.14 branches, or wait until after 1.14 branches?

[rgommers]

"it fixes a bug" is a good enough reason to bump a submodule I'd say.

[mdhaber]

This submodule bump fixes a bug, so merging.

[tylerjereddy]

The backport label has been removed because on MacOS ARM with clang-16.0.6 I saw additional test failures with this branch cherry-picked in to the release branch (not just for the new test added here).

[maresb]

Hey, great news, I just wanted to mention that I've been using a dev version of scipy from right after this merge. Previously I experienced a failure rate of roughly one in a million from my dataset. Now, after billions of invocations, I haven't seen a single failure. This is awesome, thanks so much everyone!!!