StackStorm v3.2.0 Retrospective

[arm4b]

Following the https://stackstorm.com/2020/04/30/stackstorm-v3-2-0-released/

I'd like to analyze the v3.2.0 release: what we did, what worked good and what not during the process. Make conclusions and think about actions to improve the future releases.

The good

• We finally got the v3.2.0 version out after 1 year. Yay!
• We opened the Release Mechanisms and other Release processes at https://github.com/StackStorm/discussions/wiki
• With the opened releases, part of our community really surprised us participating in the StackStorm v3.2dev pre-release Testing #6 and helped a lot, not just via reporting bugs but also fixing them.

• Release Automation wasn't that much broken, its was in OK-ish state, needed just a few fixes to st2ci and st2cd
• Majority of TSC was active as a team and helpful to load-balance:
  • getting the missing items done for 3.2.0 milestone
  • fixing the automation bugs and workflows when the release started
  • providing hotfixes for the fresh discovered issues in st2 core
• New TSC members worked as a team with the older StackStorm maintainers and accumulated more experience

Observations

• It took almost 3 months for all the release preparations, sync-ups, follow-ups and so on
• It took a lot of time to get the 3.2.0 Milestone from 85%->100%, which included just a few minor items to fix and several community-contributed PRs. They were in backlog as we failed to review them before.
• A few people can actually help with the core StackStorm/st2 codebase and drive that platform.
  • We're lucky to have members in team who drive the Orquesta and help with the st2, but with little st2 core brains and development there is little future.
  • We can't bike-shed too long near all the systems there were built around st2 core. We should find more ways to learn and improve the stackstorm/st2 itself.
• It took almost 2 weeks for the release testing StackStorm v3.2dev pre-release Testing #6. It usually took several days before.
• In general, TSC wasn't active on reporting their release testing progress. They could do better next time, giving an example to others.
• I took 1 week to run the release automation and get 3.2.0 out. This was faster and could be faster.
• Many people spent nights trying helping to get the release out. It's not normal and not sustainable.
• Only part of the TSC members were active.
• Enterprise & OSS separated releases showed some friction to get the things synced.
• Release manager was sometimes aggressive pushing the team towards v3.2.0 and follow-ups on the release items.
• We've found last-minute problems with the Javascript projects like st2web and st2chatops that weren't maintained for a long time and accumulated enough outdated dependencies that might be vulnerable. We decided to release what we have and postpone web UI/chatops fixes to v3.2.1 instead of blocking v3.2.0. Even though, st2web and st2chatops projects are still looking for their maintainer teams.

So what?

We absolutely should do a smaller releases that would include lower number of major changes and covering less blockers like many groups responsible for the different feature set. We should try to do these smaller releases more frequently.

To get the faster releases it would be great to find the ways simplifying the Release Automation and reduce the CI/CD complexity. Future mistral deprecation and rotation of OS platforms we support would help, but that's not enough.

Focus on making sure support maintenance and churn is minimal, free time on what's really important. As we got the ice broken with the first release we need consolidating that result and making it more repeatable with the next releases.

New TSC members which have still a fresh eye on the process, - please find an opportunity to report and propose the ways to simplify the things. As faster all the TSC members will conduct a new release as Release Managers, - the better it'll be for the project and team to evolve.

With new community-driven and open Governance StackStorm is good as collective effort that drives it. As maintainer/contributor team is forming and finding the ways to be successful as a group and working with community better, expect StackStorm quality to lower a bit at early stage with possibility to jump higher if we do it right.

The project can't rely on superhero approach when existing TSC Maintainers spending nights to get the things done. We need to learn working better with our community and new potential members, mentor them, share the experience (as GOVERNANCE.md suggests), cultivate more stackstorm/st2 core improvements, keep the quality bar high (important at early stage), ask for help where more contribution/maintenance work is needed, and motivate existing orgs that use StackStorm in their day-to-day production workloads to get involved.

Let's improve the things as we go and I'm looking forward for the better next releases.