Release of Stan Math 3.4

[rok-cesnovar]

Based on our schedule the next release is set to happen on the 19th of October, which means that the feature freeze will begin on the 12th of October (next Monday).

Release notes will be generated using the Github Actions job: https://github.com/stan-dev/math/actions?query=workflow%3A%22Release+scripts%22
We only need to do some copy edit afterwards (list all generalize changes together for example). If no one will volunteer for the copy-edit, I will do that on the 13th.

I would like to discuss how we handle the feature freeze. Would it be possible to do the feature freeze by merging develop at that point to master and tagging that as release candidate? We could then continue merging to develop as normally even during the feature freeze. If any issues come up with the RC during the freeze period we would make a bugfix PR to master and not develop. So master would then be RC+bugfixPRs

The reason I am asking this is that we do 4 releases a year and doing a full stop of operations on Math for a week per release means that we lose a month/year that way. And in periods of active development (like the last release for example), this can also cause huge backlogs of PRs that are waiting to be merged. And we then have PRs that are done to branches and that can get tricky really fast.

Dont think this is necessary for other repositories (Stan and Cmdstan certainly arent as active), but Math is currently in a huge transition and I think this would be worthwhile.

[wds15]

How can then the bug fixes to the RC be back merged to develop in a clean way?

[rok-cesnovar]

Would a PR merging master to develop after the actual release be considered clean?

[wds15]

Maybe. This is a question about keeping git history nice. I recall that we had long discussions over it...let's ping @seantalts for his wisdom here.

[seantalts]

doing a full stop of operations on Math for a week per release

We shouldn’t need a full stop, just to hold off on merging things to develop. Development can continue but we lose synchronization for a week each time.

Math is currently in a huge transition

Fair enough, synchronization is important in those cases.

The value of the feature freeze is that people are using and developing off the frozen branch for a while. If we change it to the proposed master merge, we lose the value of the freeze. Unless you all are aware of some people using the master branch for their work before it is released?

So perhaps it is not worth doing a feature freeze during this period of math activity? Though it seems like periods of heavy development are those that most need some additional testing. Maybe better to skip this release?

Would a PR merging master to develop after the actual release be considered clean?

It wouldn’t automatically merge cleanly, but you could resolve any conflicts by hand. Depends on what’s happened in develop since master was pulled off.

[SteveBronder]

Maybe better to skip this release?

I wouldn't mind taking it to a vote. If we are in the middle of a bunch of stuff not doing one quarterly release is not a bad thing. Though it would also be very nice to expose @andrjohns 's new scalar binary operations so I'm very 50/50 (though they are exposed if people are using cmdstan develop). It feels like we set the release cycle quarterly because we were bad at actually putting down a date for the release. Skipping a release seems find though it does teeter towards "Well we will release when this one feature is done", which is why we made the release's quarterly lol. If we put it to a vote I'd phrase it as "We will cancel one quarterly release and next release can not be rejected by a vote".

Would a PR merging master to develop after the actual release be considered clean?

It wouldn’t automatically merge cleanly, but you could resolve any conflicts by hand. Depends on what’s happened in develop since master was pulled off.

I think what Eigen does is rebase branches before merging where they seperate out the bug fix and the feature. Then you can just cherry pick from each. Rebasing is very annoying tho

[bbbales2]

I want to release! I like the regular releases. We must beckon in the fall season at Stan!

Now that I think about it, we should do 2.26 two months from now so Stan has a Christmas release and then we can be properly aligned with the seasons.

Re: Master as release candidate. Sure, or a release_2.25 branch, whatever makes life easier!

[rok-cesnovar]

I also want to do the release.

Other than the vectorized binary operations, OpenCL GLM support for x and y being parameters and a few other things this is more of a transitional release. Its worth it to only release for that and if there are bugs that occurred during the backend refactor we will get feedback/bug reports earlier which is nice.

Its probably to quick to make a decision now for this release, I started thinking about this too late. The clean git history is important.

[SteveBronder]

That was a quick vote then let's do it!

[bbbales2]

this is more of a transitional release

I guess it is transitional, but I wouldn't underestimate the changes here -- all of reverse mode is changing! New autodiff for the distributions, new var type, new autodiff for basically everything that had it previously (scroll through the benchmarks -- though I'm missing a few functions and need to update it again: https://bbbales2.shinyapps.io/math-shiny-perf/).

That's gonna effect everyone. Hopefully models just go faster (the benchmarks look good). It'll be good to see how the cookie crumbles on it though.

[rok-cesnovar]

There is a ton of great stuff. I meant more in the sense that none of these long-term project are quite done yet.

The revamped distributions and generalization are still missing a few functions, though that is really really close. 2.26 should have all of them plus returning expressions. The var_value stuff will not land in 2.25 (maybe a function or 2 if i can get the stanc3 parts in, I am a bit scared to get stuff in the last week though) but will be in Math 3.4 that is true. The new autodiff with reverse callback is slick a hell but also not there for all functions just yet. But will be for 2.26 (or at least closer). That is what I meant by transitional.

This will all be amazing, I am not underestimating that! Please dont get me wrong, I am very hyped for all of these fun stuff.

[rok-cesnovar]

Basically, by transitional I meant that we cant release and say "Stan Math 3.4: var matrices reloaded", "Stan Math 3.4: expression expression expressions" or "Stan Math 3.4: calling back smoother" just quite yet, but we (probably) will for 2.26.

[seantalts]

How much of the reverse mode et al will be in this release for Stan users? Just curious as a Stan user myself these days, haha. Very exciting. Also if there's something I can try to hammer out on the compiler before the freeze that would help, let me know.

The clean git history is important.

I didn't mean to suggest the git history would look worse afterwards if we did another way - I think it'll be fine. I know because I've used something similar to the model you're talking about before at a few companies I've worked at.

My only argument for the current feature freeze system is that freezing the branch that everyone develops from is a cheap way to get some additional testing before it goes out to users. We went this route because when we introduced the release deadlines, folks understandably started submitting large chunks of work hours or minutes beforehand, and our automated testing coverage isn't 100% and we had some bugs slip into the release. There are probably multiple solutions to this problem, feature freezes just seemed like an easy cheap one at the time and I'm not married to it forever.

I think the "git flow" model where you tag master and build bug-fixes on master and then backport to develop is really excellent for companies with 1. great automated testing, 2. development staff on-call if a bug is discovered and 3. the ability to continuously release, usually on the web. Our situation releasing desktop software looks pretty different to me.

[bbbales2]

I meant more in the sense that none of these long-term project are quite done yet

Yeah, but I've been surprised by how much stuff has changed just with the mat<var> cleanup. The thing var_value was going to do that I was most excited about was fix matrix vector multplies and make vectorization worth it.

Unexpectedly (for me) the matrix vector multiplies are way more efficient with this (like weirdly faster). Our implementation must have been doing something wrong that got fixed in the cleanup since the cleaned up version is doing all the same stuff algorithmically. It looks like the vectorized stuff is a tad bit faster too (#2121) (edit: actually, I don't think "tad bit" is appropriate here -- speeding up stuff like this is hard and it is no questions absolutely faster).

Also the stuff Tadej did with the distributions is pretty radical. I need to update the benchmarks to all he's done, but if anyone is doing gumbels, that one is nearly 3x faster for big N.

but also not there for all functions just yet

If I missed anything it was a mistake on my part: #2019 vrrm vrrrm gotta go get that green lighted

we cant release and say

I'm stuck on the var<mat> -> varmat -> varmint connection. It should be "2.26, With Varmint Tech" (and then we have a drawing of a rat with a calculator or something).

[bbbales2]

"Introducing varmint tech in 2.26"

[rok-cesnovar]

If I missed anything it was a mistake on my part: #2019 vrrm vrrrm gotta go get that green lighted

Oh, is that PR going to move everything to reverse_pass_callback? I did not know that! If that gets in for the release that is very nice!

I'm stuck on the var -> varmat -> varmint connection. It should be "2.26, With Varmint Tech" (and then we have a drawing of a rat with a calculator or something).

😃

Also the stuff Tadej did with the distributions is pretty radical.

Oh yes, those are very nice.

How much of the reverse mode et al will be in this release for Stan users?

As far as I understand (but please correct me Ben, Steve or Tadej):

• move to reverse_pass_callback: Based on what @bbbales2 is saying, its done or will be done once Simplifying existing autodiff to prepare for adding var_value<Matrix<T, R, C>> support to rev/fun #2019 gets in. As Tadej says, the reverse_pass_callback reduces the lines of code, reducing the chance of doing something suboptimal :) I like that quote.
• var<mat>: merged in Math for multiply, elt_multiply, softmax and dot_product. Missing stanc3 side of things*.
• expressions for /prim: The list of functions that do not accept expressions is shrinking daily. Once all the open PRs are merged, we will be down to:

algebra_solver, algebra_solver_newton
binomial
block
columns_dot_product, columns_dot_self
csr_extract_u, csr_extract_v, csr_extract_w
csr_matrix_times_vector, csr_to_dense_matrix
dirichlet
distance
dot_product
if_else
lambert_w0, lambert_w1
log_determinant
map_rect
matrix_exp_multiply
multiply_lower_tri_self_transpose
rows_dot_product
scale_matrix_exp_multiply
sd
uniform
von_mises
weibull
wiener
wishart

All the generalized distributions were optimized and now use expressions internally which is a huge win. The list of Stan signatures that are not tested for expressions is here: https://github.com/stan-dev/math/blob/develop/test/expressions/stan_math_sigs_exceptions.expected

• OpenCL var<matrix_cl> is supported for GLMs, multiply and sum, with cholesky decompose PR open. stanc3 GLM support for this is already in stanc3. A PR for the rest and for combining them is still in the works (mostly combining them). I am hoping to maybe get it ready by Thursday, but am a bit scared of rushing it. Will see.

*I started working on that but the way it works right now we would have to check whether a given matrix is ever indexed. Since that requirement is going away in the next few weeks and the low number of functions currently supported (causing us to move var<mat> to mat<var> or back too often) that will wait for the next release I guess.