docs(handbook): Introduce maintenance description

[ppawlowski]

Description

This pull request introduces Infrastructure Maintenance page to the handbook. The goal of this page is to provide high level documentation on steps required to peroform successfull maintenance of various components of FlowFuse Cloud infrastructure.

Related Issue(s)

https://github.com/FlowFuse/CloudProject/issues/624

Checklist

• I have read the contribution guidelines
• I have considered the performance impact of these changes
• Suitable unit/system level tests have been added and they pass
• Documentation has been updated

[gstout52]

@ZJvandeWeg

[ZJvandeWeg]

How can they prevent this? The way our infra is set up influences all this, and we never get this from Heroku et al. Why do we get this from FlowFuse?

[hardillb]

It's fundamental to how kubernetes works.

You can only have a version mismatch of +- 2 (e.g. 1.24 to 1.26) between the core kubernetes api and the version of the kubelet managing the node the pods are running on.

So to upgrade kubernetes version you upgrade the core api, then you need to create new nodes, and then migrate the running pods to those new nodes. The migration requires stopping the instance on the old node and starting it on the new version.

In theory we could start the instance, but that does end up with 2 instances running for a short while, which requires planning in the flow design (around caching or state management)

[ZJvandeWeg]

@hardillb My point is that other compute providers just restart, even cloud providers. Somehow we don't and send emails regularly to customers asking them to restart instances, and we don't even tell them which ones.

[knolleary]

Some updates based on negotiated terms with a customer (and a couple typos I'll self-merge)

[ZJvandeWeg]

Let's describe how this process of commucation is done. GitHub issue seems best with a mention to @Esmewieringa, @MichaelBDavis, @gstout52 and myself.

[ppawlowski]

As far as I understood there is such document somewhere. @gstout52 can you shed some light, please?

[gstout52]

@ppawlowski there isn't a doc describing this process. Here are pieces that should be included in the handbook regarding how communications are sent to customers as well as the internal work necessary to accomplish this:

• Engineering prepares a list of customers who need to receive the email. This email should include team owner email addresses, team names, and the names of instances that need to be restarted. The intent is that it should be very easy for the customer to quickly identify what is needed from them.
• Email copy should be drafted and approved by both Engineering and the Customer Department. Email copy should aim to be informative while being accessible.
• Email copy should include the timeframe of the migration/restart, and this timeframe should be a brief window (for example, a specific two-hour time period).
• If possible, the email should include the recipient's first name.

If at all possible, avoid emailing customers about behind-the-scenes work. When there is a way to provide excellent service within our stated SLA, just take the needed action.

[ZJvandeWeg]

@ppawlowski Handbook in combination with a GH issue template seems to be the way to go. Note there there's more and more requirements by customers for scheduled downtime and the company needs to properly manage this -- best way to make sure we do things the right way is to make a process and follow it.