Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(handbook): Introduce maintenance description #2944

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

docs(handbook): Introduce maintenance description #2944

Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
d0739c1
Introduce maintenance description
ppawlowski Jan 24, 2025
a3f4b18
Clarify when message should be sent
ppawlowski Jan 24, 2025
2e5d096
Update src/handbook/development/ops/maintenance.md
ppawlowski Feb 6, 2025
7cbb80b
Update src/handbook/development/ops/maintenance.md
knolleary Mar 11, 2025
592bf36
Update src/handbook/development/ops/maintenance.md
knolleary Mar 11, 2025
b498cda
Update src/handbook/development/ops/maintenance.md
ppawlowski Mar 11, 2025
bc9c19f
Update src/handbook/development/ops/maintenance.md
ppawlowski Mar 11, 2025
0352731
Update src/handbook/development/ops/maintenance.md
ppawlowski Mar 11, 2025
151b9e6
Merge branch 'main' of github.com:FlowFuse/website into docs-handbook…
ppawlowski Mar 11, 2025
371cb74
Update src/handbook/development/ops/maintenance.md
ppawlowski Mar 11, 2025
59b4c57
Adjust information about notifcations
ppawlowski Mar 11, 2025
fcbc473
Update src/handbook/development/ops/maintenance.md
knolleary Mar 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions src/handbook/development/ops/maintenance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
navTitle: Infrastructure Maintenance
---

Periodically, FlowFuse performs maintenance on FlowFuse Cloud resources. This page describes how we manage and execute maintenance tasks around the infrastructure.

# Maintenance scope

The maintenance most often involves updates to the following resources:
* underlying cluster hosts
* Kubernetes versions
* databases
* networking components


# Maintenance process

## Planning

The maintenance process is initiated by the Operations Team. The process is planned in advance and communicated to the customers. The maintenance window is scheduled to minimize the impact on the team and customers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The maintenance process is initiated by the Operations Team. The process is planned in advance and communicated to the customers. The maintenance window is scheduled to minimize the impact on the team and customers.
The maintenance process is initiated by the Operations Team.
A new [Infrastructure task](https://github.com/FlowFuse/CloudProject/issues/new?template=infra-task.yml) is created to co-ordinate all activities regarding the task - including the technical tasks and communications with customers.
The maintenance window is scheduled to minimize the impact on the team and customers.


Where the maintenance action will cause disruption to customers, it must be scheduled with the following considerations:
* at least 45 days notice should be provided to customers
* not during the working week
* maintenance should not overlap with [scheduled release](../releases/process.md) week
## Customers communication

Once the maintenance date is set, customers should be informed about the upcoming maintenance. This must be done in co-ordination with Customer team. The communication should include:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's describe how this process of commucation is done. GitHub issue seems best with a mention to @Esmewieringa, @MichaelBDavis, @gstout52 and myself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understood there is such document somewhere. @gstout52 can you shed some light, please?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ppawlowski there isn't a doc describing this process. Here are pieces that should be included in the handbook regarding how communications are sent to customers as well as the internal work necessary to accomplish this:

  • Engineering prepares a list of customers who need to receive the email. This email should include team owner email addresses, team names, and the names of instances that need to be restarted. The intent is that it should be very easy for the customer to quickly identify what is needed from them.
  • Email copy should be drafted and approved by both Engineering and the Customer Department. Email copy should aim to be informative while being accessible.
  • Email copy should include the timeframe of the migration/restart, and this timeframe should be a brief window (for example, a specific two-hour time period).
  • If possible, the email should include the recipient's first name.

If at all possible, avoid emailing customers about behind-the-scenes work. When there is a way to provide excellent service within our stated SLA, just take the needed action.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ppawlowski Handbook in combination with a GH issue template seems to be the way to go. Note there there's more and more requirements by customers for scheduled downtime and the company needs to properly manage this -- best way to make sure we do things the right way is to make a process and follow it.


* the scope of the maintenance
* the date and time of the maintenance
* expected duration of the maintenance
* self-care actions which customers can/should take before the maintenance date
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can they prevent this? The way our infra is set up influences all this, and we never get this from Heroku et al. Why do we get this from FlowFuse?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fundamental to how kubernetes works.

You can only have a version mismatch of +- 2 (e.g. 1.24 to 1.26) between the core kubernetes api and the version of the kubelet managing the node the pods are running on.

So to upgrade kubernetes version you upgrade the core api, then you need to create new nodes, and then migrate the running pods to those new nodes. The migration requires stopping the instance on the old node and starting it on the new version.

In theory we could start the instance, but that does end up with 2 instances running for a short while, which requires planning in the flow design (around caching or state management)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hardillb My point is that other compute providers just restart, even cloud providers. Somehow we don't and send emails regularly to customers asking them to restart instances, and we don't even tell them which ones.

* other relevant details to help customers to prepare for potential restart of their Node-RED instances

E-mail template for maintenance communication can be found in the [FlowFuse Cloud Maintenance Communication document](https://docs.google.com/document/d/1Pkd0qifjgs7xv96hfQ5OHnvX5n_WPHyA0AvPd9UUPL0). Adjust the template accordingly.

The notification message should be sent to all FlowFuse Cloud Team Owners who have active instances in the FlowFuse Cloud and who are affected by the maintenance. To get the contact list, please refer to the [internal documentation](https://docs.google.com/document/d/1s0mXpuuPKl-1U-YVzoTEGBhllUd-UPJjFXTqa0O481c/#heading=h.hing87m0bsmu).

We should sent notification at least twice - 45 days before the maintenance and a reminder one week before the maintenance day.

Before sending the initial message, consult with the Operations team to ensure that the information is up-to-date and infrastructure is ready to handle self-care maintenance by customers.
Comment on lines +36 to +42
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
E-mail template for maintenance communication can be found in the [FlowFuse Cloud Maintenance Communication document](https://docs.google.com/document/d/1Pkd0qifjgs7xv96hfQ5OHnvX5n_WPHyA0AvPd9UUPL0). Adjust the template accordingly.
The notification message should be sent to all FlowFuse Cloud Team Owners who have active instances in the FlowFuse Cloud and who are affected by the maintenance. To get the contact list, please refer to the [internal documentation](https://docs.google.com/document/d/1s0mXpuuPKl-1U-YVzoTEGBhllUd-UPJjFXTqa0O481c/#heading=h.hing87m0bsmu).
We should sent notification at least twice - 45 days before the maintenance and a reminder one week before the maintenance day.
Before sending the initial message, consult with the Operations team to ensure that the information is up-to-date and infrastructure is ready to handle self-care maintenance by customers.
The e-mail template for all maintenance communication can be found in the [FlowFuse Cloud Maintenance Communication document](https://docs.google.com/document/d/1Pkd0qifjgs7xv96hfQ5OHnvX5n_WPHyA0AvPd9UUPL0). Create a copy of the template and link to from the CloudProject issue.
The notification message should be sent to all FlowFuse Cloud Team Owners who have active instances in the FlowFuse Cloud and who are affected by the maintenance. To get the contact list, please refer to the [internal documentation](https://docs.google.com/document/d/1s0mXpuuPKl-1U-YVzoTEGBhllUd-UPJjFXTqa0O481c/#heading=h.hing87m0bsmu).
The Customer team will handle sending the email once the content is agreed with engineering and the target list of customers has been provided.
The first notification should sent at least 45 days before the event. A reminder should be sent one week prior to the event.


## Infrastructure preparation

All preparations should be done before the maintenance window by the Operations team.

### FlowFuse Cloud cluster

#### Upgrading Kubernetes version - control plane

The control plane upgrade does not require a maintenance to be scheduled - the process is automated and can be executed at any time. It does not affect the running workloads.
The process is described in the [cluster upgrade documentation](https://github.com/FlowFuse/CloudProject/blob/main/UPGRADE.md#cluster-upgrade).

#### Upgrading Kubernetes version - worker nodes

The worker nodes upgrade process is described in the [cluster upgrade documentation](https://github.com/FlowFuse/CloudProject/blob/main/UPGRADE.md#node-groups-upgrade).
High-level upgrade steps are:
* create a new node group(s) with the desired Kubernetes version
* prohibit autoscaling on the old node group(s)
* cordon the old node group(s) - this step should be performed shortly before sending the maintenance notification
See documentation for more details.

## Maintenance day

All maintenance activities should be communicated in the `#ops-cloud` slack channel.

### Kubernetes worker nodes upgrade

On the maintenance day, the Operartions Team should perform the following steps to migrate workloads to the new node group(s):
* selectively restart workloads still running on old node group(s).
* monitor the workloads and ensure that:
* they were scheduled on correct node group
* Node-RED flows started correctly - take corrective action (e.g., redeployment or log analysis) if issues occur

Once all workloads are migrated to the new node group(s), the old one can be safely [terminated](https://docs.aws.amazon.com/eks/latest/userguide/delete-managed-node-group.html#eksctl-delete-managed-nodegroup).
Information about the completion of migration should be shared with the team.