DOC: prepare 1.13.0 release notes

[tylerjereddy]

• prepare the release notes for SciPy 1.13.0, which is an out-of-band release primarily aimed at supporting NumPy 2.0.0

[skip ci] [skip circle]

TODO:

• plan for release candidates vs. just putting 1.13.0 out as soon as NumPy 2.0.0 is up on PyPI and the wheel builds are passing in our CI (or if NumPy will do RCs or whatever that guarantee stability..)
• adjustments to pyproject.toml (etc.) for the "new way" to build against NumPy 2.0.0 and have backwards-compat to older NumPy that way (if any adjustments are still needed...)
• any NumPy 2.0.0 pre-release compat issues left? yes, MAINT: maintenance/2.0.x branch binary compat issue on SciPy main numpy/numpy#26060, I suggested they revert but will deal with after branching probably...
• .mailmap adjustments to improve author list
• custom wording beyond what I've done for this out-of-band support release
• scan over the merged enhancement PRs and release notes wiki again before branching, etc.
• release notes highlights, maybe sparse and stats
• gradually reactivate CI, but for now skip completely

[h-vetinari]

prepare the release notes for SciPy 1.13.0, which is an out-of-band release primarily aimed at supporting NumPy 2.0.0

Do we really want to bump the minor number here, "just" for supporting NumPy 2.0? I guess the main argument relates to the volume of changes necessary to adapt to API-changes in 2.0. I can't tell how big that volume would be, and if we'd be comfortable with putting this into a 1.12.1.

I ask primarily because in terms of deprecation announcements, we've been pretty clearly communicating "will be removed in 2 releases" warnings, with the expectation that this would give users about a year to adapt. If we're introducing another release in between this is either going to mess up our announcements or quite substantially shorten the timespan from deprecation to execution.

In case we stay with the choice to release 1.13 out of band, we should then at least consider increasing all the current deprecation warnings by one (before 1.12 GA), though that'd still leave the various deprecations we already executed for 1.13 on a very accelerated schedule (unless we revert them).

CC @rgommers @j-bowhay

[h-vetinari]

FWIW, I'd prefer if we backport the relevant changes for numpy 2.0 to 1.12 rather than release 1.13 out-of-band.

[lucascolley]

the main issue there is that there are still breaking changes which haven't happened yet in NumPy for 2.0.

[rgommers]

It'll be way better for debugging potential issues if there's a clear split here with a new minor release. There's a lot that changes in 2.0, and also a lot of fixes for 2.0 went into main and were not backported because of that clear split - we'd have to go through everything that was merged and figure out what to do. I'm -1 on changing that plan now. Deprecation are not that important, and not a good reason to change the plan now.

we've been pretty clearly communicating "will be removed in 2 releases" warnings .... If we're introducing another release in between this is either going to mess up our announcements

We've been overdoing it on policy I think. It's completely fine to remove something in 3 releases after deprecations. That should not require even a change in message - those messages are mistakes then if they were that specific. It's a minimum of 2 releases.

[h-vetinari]

We've been overdoing it on policy I think. It's completely fine to remove something in 3 releases after deprecations. That should not require even a change in message - those messages are mistakes then if they were that specific. It's a minimum of 2 releases.

I disagree on that. It's taken a fair amount of effort to get away from the previous "it's deprecated, but not removed for a long time", which just breeds complacency because we never followed through. So now we give users actionable warnings that say "this function will disappear in version 1.X". I continue to think this is useful both from a communication and a maintenance standpoint.

Since we started doing this in the 1.8/1.9 timeframe, we've had a very reliable 6 month release cadence, so the issue at hand never came up. I accept that numpy 2.0 is an exceptional event and that an out-of-band release might be the best choice. But then we just need to our messaging (good we caught this before 1.12 GA...), not the way we've been doing deprecations.

[rgommers]

So now we give users actionable warnings that say "this function will disappear in version 1.X".

This is too absolute. It's a good default, but it should always be possible to leave something in for longer if there is a reason not to remove it. If new deprecation messages are written to give zero wiggle room to a removal schedule, that's a mistake I think - it'd be better to say "is likely to be removed in ...".

[h-vetinari]

it'd be better to say "is likely to be removed in ...".

Clearly we don't have a crystal ball, and we've missed announced removals or changed the announced version in the past. It remains "best effort" as opposed to "set in stone", but in this case, we could simply adapt the warning messages that we're releasing for 1.12?

The more annoying question is whether we want to undo all the deprecations that were already executed on main (with the expectation of a mid-2024 release of 1.13)

[lucascolley]

The more annoying question is whether we want to undo all the deprecations that were already executed on main (with the expectation of a mid-2024 release of 1.13)

I think that this seems slightly unnecessary given that, assuming the NumPy release continues as speculated, 1.13 will still be ~9 months after 1.11. Sure, this will be unexpected for anyone who is keeping track of our normal release schedule, but I think it still gives a reasonable amount of time to adapt.

[rgommers]

we could simply adapt the warning messages that we're releasing for 1.12?

Sure, if you want to do that that seems fine.

Please consider following https://numpy.org/neps/nep-0023-backwards-compatibility.html#implementing-deprecations-and-removals for future deprecations though, it works pretty well in NumPy.

The more annoying question is whether we want to undo all the deprecations that were already executed on main (with the expectation of a mid-2024 release of 1.13)

It's the list in gh-15765 for 1.13, right? 6 PRs so far? I dunno, either way seems fine. Given that many packages will do a new numpy 2.0-compatible release, maybe it's not too much of a problem here to do these ones a bit faster.

[ilayn]

I have a few hours worth work remaining for the specfun rewrite, 5 failing tests remaining. Mostly typo finding etc. since these are (lots of) small chunks of functions.

I didn't know 1.13 was the NumPy 2.0 release so I can actually push it a bit more and finish it off if there is time. I'm wondering how much time are we talking about until this starts flying, what do you say @tylerjereddy ?

[tylerjereddy]

I'm wondering how much time are we talking about until this starts flying, what do you say @tylerjereddy ?

I think late January/early February is what I'd heard in the last month or two from NumPy folks. They do have a 2.0 progress board: https://github.com/orgs/numpy/projects/9. I believe world events did cause some delays to original plans, and I don't know if they'll push a bit later than that now.

It seems likely you'd finish up a few hours of work before 1.13.0 gets released, and I don't even have 1.12.0 final out yet.

Anyway, part of the point of opening this was to advertise it a bit more broadly, so hopefully we find a solution that doesn't cause too much friction. The original 1.12.0 announcement email to mailing list on Nov. 11/2023 did mention the plan to do 1.13.0 out of band without objections, although if others are like me they don't always monitor that list.

[rgommers]

It's getting close, but there's always some last-minute hiccups. So if the plan is 31 January, it'll probably be sometime in the ~10 days after that.

[ilayn]

Ah OK, that's plenty sans reviewing.

[tylerjereddy]

@mdhaber @h-vetinari I think I saw some emails/queries about scheduling. My plan was to do this for 1.12.0 scheduling: try to do 1.12.0 final on Jan. 19th/20th, which is one week after RC2 happened. If you think I need to delay that to allow the deprecation wording to get relaxed or whatever, let me know.

[h-vetinari]

My plan was to do this for 1.12.0 scheduling: try to do 1.12.0 final on Jan. 19th/20th, which is one week after RC2 happened. If you think I need to delay that to allow the deprecation wording to get relaxed or whatever, let me know.

My PR (following a quick analysis that no-one objected to) is here: #19892. I checked that it cherry-picks cleanly to maintenance/1.12.x.

[ilayn]

@tylerjereddy can we ask for at least a quick 1.13.1 because we were under the impression that this would be a regular release. If I missed the convo, my apologies in advance. There will be quite some bug reports I am anticipating for the Fortran rewrites but in 10 days, we might not clean up the low hanging fruit.

NumPy announcement should be a bit more louder in my opinion. We should probably hit all the relevant geek squad circles.

[tylerjereddy]

can we ask for at least a quick 1.13.1

Yes, I have some funding support to do releases and happy to do what I can to help. For urgent release problems Ralf usually helps me get unstuck pretty fast too. Of course, there's also the conda-forge part so I may have to be cautious not to dump too much on others :)

[ilayn]

I can at least promise that our fixes will be way less sophisticated than NumPy 2.0 issues.

[tylerjereddy]

Suggested timeline from NumPy team meeting this morning was ~late February.

[tylerjereddy]

Some timeline updates from upstream are here: numpy/numpy#24300 (comment)

I will start updating the release notes "soon" of course (I keep bumping the calendar reminder, but really do need to start back at it soon). Chuck is working on the beta release process as I write this.

[rgommers]

@tylerjereddy it may be worth announcing a timeline on the mailing list, to give people a chance to get some last things in for the release? It usually motivates getting something over the finish line.

adjustments to pyproject.toml (etc.) for the "new way" to build against NumPy 2.0.0 and have backwards-compat to older NumPy that way (if any adjustments are still needed...)

I suggest keeping this PR for release notes only. Re pyproject.toml, yes we need a bunch of changes - I'm having a look at that now.

any NumPy 2.0.0 pre-release compat issues left?

I think we're all good here, since we have working wheel builds with NumPy main, which has branched now.

[rgommers]

See https://github.com/scipy/scipy/compare/main...rgommers:scipy:update-deps-for-1.13?expand=1 for changes I think we need - I can open a PR after maintenance/1.13.x is created.

[tylerjereddy]

Great, thanks. Yes, this PR is release notes only. I made a calendar reminder to send something to the mailing list Monday evening, if the NumPy beta is out and I have a tentative plan.

[tylerjereddy]

This is the tentative/proposed schedule I sent to the mailing list for feedback:

• branch maintenance/1.13.x: March 17/2024
• SciPy 1.13.0rc1 release: March 18/2024
• SciPy 1.13.0rc2 release: March 25/2024
• SciPy 1.13.0 final release: March 31/2024

[mdhaber]

Suggested change

	- `scipy.stats.rankdata` has been vectorized, improving its performance and the
	performance of hypothesis tests that depend on it.
	- ``stats.mannwhitneyu`` should now be faster due to a vectorized statistic
	calculation.
	- `scipy.stats.mood` now has ``nan_policy`` and ``keepdims`` support.
	- `scipy.stats.shapiro`, `scipy.stats.normaltest`, `scipy.stats.skewtest`,
	`scipy.stats.kurtosistest`, and `scipy.stats.kstest` have gained ``axis``,
	``nan_policy`` and ``keepdims`` support.
	- `scipy.stats.boxcox_normmax` has gained a ``ymax`` parameter to allow user
	specification of the maximum value of the transformed data.
	- `scipy.stats.wasserstein_distance_nd` was introduced to compute the Wasserstein-1
	distance between two N-D discrete distributions.
	- Vectorized calculations with `scipy.stats.wilcoxon`, `scipy.stats.mannwhitneyu`,
	and `scipy.stats.rankdata` are faster.
	- The ``method`` parameter of `scipy.stats.wilcoxon` and `scipy.stats.mannwhitneyu`
	now accepts instances of `scipy.stats.PermutationMethod` for computing exact
	or randomized *p*-values, even in the presence of ties and/or zeros.
	- `scipy.stats.boxcox_normmax` now accepts a ``ymax`` parameter, which allows users
	to specify the maximum allowable magnitude of the transformed data when
	``method='mle'``.
	- The ``fit`` methods of `scipy.stats.gamma` (with ``method='mm'``) and
	`scipy.stats.loglaplace` are faster and more reliable.
	- Moment calculations of `scipy.stats.powerlaw` are faster and more accurate.
	- The ``statistic`` parameter of `scipy.stats.goodness_of_fit` now accepts custom
	callables.
	- Support for ``axis``, ``keepdims``, and/or ``nan_policy`` keyword arguments has
	been added or improved for the following functions: `scipy.stats.f_oneway`,
	`scipy.stats.alexandergovern`, scipy.stats.shapiro`, `scipy.stats.normaltest`,
	scipy.stats.skewtest`, scipy.stats.kurtosistest`, scipy.stats.friedmanchisquare`,
	scipy.stats.brunnermunzel`, and `scipy.stats.mood`.

@dschmitz89 If there are any PRs you reviewed/merged that aren't listed here, please feel free to add them.

@tylerjereddy IIRC in the past, we have drafted these in the Wiki and the PR gets created from that shortly before release. Will we be doing this differently going forward?

[mdhaber]

When someone is editing locally, these scipy.stats sub-categories can be removed. Also, if you're willing to add to the "Other Changes" section :

- The second argument of `scipy.stats.moment` has been renamed to `order` while maintaining backward compatibility.

I'd appreciate it!

[mdhaber]

Suggested change

	`scipy.interpolate` improvements
	`scipy.integrate` improvements
	==============================
	- The `terminal` attribute of event functions passed to `scipy.integrate.solve_ivp` may now be a positive integer: integration stops after `terminal` events have occurred.
	`scipy.interpolate` improvements

[tylerjereddy]

Yeah, I've already got this one locally. I guess we'll see how things compare when I push up in a bit.

[mdhaber]

Suggested change

- `scipy.stats.invwishart` ``rvs`` and ``logpdf`` are now faster.

[tylerjereddy]

I've been working on the release notes locally for a few hours, so I'll need to take a look at how things mesh after I push up.

[tylerjereddy]

Ok, I've pushed up what I have locally for now, to make it easier for folks to see where things stand. Probably some of Matt's stats suggestions will be better than what I drafted, though I think I've caught the essence of most changes.

Highlights and .mailmap updates still need some attention of course.

Edit: doc build is passing locally with python dev.py doc -j 1, but skipping CI here for now.

[tylerjereddy]

To address one point above, the wiki was fairly sparsely populated with updates this time around, probably because of the out-of-band timing, but I don't have any plans to change the process, I still updated from the wiki last night, then scanned through the merged PRs manually.

I'm working on a few more cleanups now, then I'll see if I can find consensus with the changes from Matt above.

[tylerjereddy]

I pushed in some updates to .mailmap, highlights section, general description (OpenBLAS version bump for wheels, etc.), and fusing in some of Matt's stuff + adding him as a co-author on commit.

The rendered version of the docs looks fairly pleasing locally, I'll let CI do a docs-only pass now.

[tylerjereddy]

I also don't plan to allow numpy/numpy#26060 to block us from branching. I'll backport a shim after branching if one is needed on our end, though I'd prefer if NumPy didn't change a public header struct size at this stage of the process.

[tylerjereddy]

Alright, Matt, Evgeni and others can suggest more release notes improvements after I branch, but the docs CI is happy here, and the render of the new release notes seems "ok."

I'm going to self-merge as branching is imminent.