Skip to content

Discourage third party software on the upstream cluster #1731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This guide contains our recommendations for running the Rancher server, and is i

### Recommended Architecture and Infrastructure

Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server on a high-availability Kubernetes cluster.
Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server for a production installation.

### Deployment Strategies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,34 @@ If you are installing Rancher in a vSphere environment, refer to the best practi

When you set up your high-availability Rancher installation, consider the following:

### Run Rancher on a Separate Cluster
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
### Minimize Third-Party Software on the Upstream Cluster

We generally recommend running Rancher on a dedicated cluster, free of other workloads, to avoid potential performance and compatibility issues.

Rancher, especially when managing a growing number of clusters, nodes, and workloads, places a significant load on core Kubernetes components like `etcd` and `kube-apiserver` on the upstream cluster. Third-party software can interfere with the performance of these components and Rancher, potentially leading to instability.

Furthermore, third-party software can functionally interfere with Rancher. To minimize compatibility risks, deploy only essential Kubernetes system components and Rancher on the upstream cluster.

The following applications and components generally do not interfere with Rancher or the Kubernetes system:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs, CPIs, CSIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)
* the [Harbor](https://goharbor.io/) container registry

Note that each of these components has its own minimum resource requirements, which must be met in addition to Rancher's.

Container registries, in particular, can require significant bandwidth for serving images. Ensure sufficient bandwidth is available, ideally reserved using Quality of Service (QoS) mechanisms, for Rancher.

For high-scale deployments, consider dedicating separate nodes to non-Rancher software using [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to minimize interference.

The following software can interfere with Rancher performance at scale and is therefore discouraged on the upstream cluster:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))

### Make sure nodes are configured correctly for Kubernetes
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,23 +23,7 @@ When scaling up Rancher, one typical bottleneck is resource growth in the upstre

### Minimizing Third-Party Software on the Upstream Cluster

Running Rancher at scale can put significant load on internal Kubernetes components, such as `etcd` or `kubeapiserver`. Issues may arise if third-party software interferes with the performance of those components or with Rancher.

Every third-party piece of software carries a risk of interference. To prevent performance issues on the upstream cluster, you should avoid running any other apps or components, beyond Kubernetes system components and Rancher itself.

Software in the following categories generally won't interfere with Rancher or Kubernetes system performance:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)

On the other hand, the following software are found to interfere with Rancher performance at scale:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))
Recommendations outlined in the [general Rancher recommendations](./tips-for-running-rancher.md#minimize-third-party-software-on-the-upstream-cluster) are particularly important in a high scale context.

### Managing Your Object Counts

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This guide contains our recommendations for running the Rancher server, and is i

### Recommended Architecture and Infrastructure

Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server on a high-availability Kubernetes cluster.
Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server for a production installation.

### Deployment Strategies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,34 @@ If you are installing Rancher in a vSphere environment, refer to the best practi

When you set up your high-availability Rancher installation, consider the following:

### Run Rancher on a Separate Cluster
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
### Minimize Third-Party Software on the Upstream Cluster

We generally recommend running Rancher on a dedicated cluster, free of other workloads, to avoid potential performance and compatibility issues.

Rancher, especially when managing a growing number of clusters, nodes, and workloads, places a significant load on core Kubernetes components like `etcd` and `kube-apiserver` on the upstream cluster. Third-party software can interfere with the performance of these components and Rancher, potentially leading to instability.

Furthermore, third-party software can functionally interfere with Rancher. To minimize compatibility risks, deploy only essential Kubernetes system components and Rancher on the upstream cluster.

The following applications and components generally do not interfere with Rancher or the Kubernetes system:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs, CPIs, CSIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)
* the [Harbor](https://goharbor.io/) container registry

Note that each of these components has its own minimum resource requirements, which must be met in addition to Rancher's.

Container registries, in particular, can require significant bandwidth for serving images. Ensure sufficient bandwidth is available, ideally reserved using Quality of Service (QoS) mechanisms, for Rancher.

For high-scale deployments, consider dedicating separate nodes to non-Rancher software using [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to minimize interference.

The following software can interfere with Rancher performance at scale and is therefore discouraged on the upstream cluster:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))

### Make sure nodes are configured correctly for Kubernetes
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,23 +23,7 @@ When scaling up Rancher, one typical bottleneck is resource growth in the upstre

### Minimizing Third-Party Software on the Upstream Cluster

Running Rancher at scale can put significant load on internal Kubernetes components, such as `etcd` or `kubeapiserver`. Issues may arise if third-party software interferes with the performance of those components or with Rancher.

Every third-party piece of software carries a risk of interference. To prevent performance issues on the upstream cluster, you should avoid running any other apps or components, beyond Kubernetes system components and Rancher itself.

Software in the following categories generally won't interfere with Rancher or Kubernetes system performance:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)

On the other hand, the following software are found to interfere with Rancher performance at scale:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))
Recommendations outlined in the [general Rancher recommendations](./tips-for-running-rancher.md#minimize-third-party-software-on-the-upstream-cluster) are particularly important in a high scale context.

### Managing Your Object Counts

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This guide contains our recommendations for running the Rancher server, and is i

### Recommended Architecture and Infrastructure

Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server on a high-availability Kubernetes cluster.
Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server for a production installation.

### Deployment Strategies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,34 @@ If you are installing Rancher in a vSphere environment, refer to the best practi

When you set up your high-availability Rancher installation, consider the following:

### Run Rancher on a Separate Cluster
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
### Minimize Third-Party Software on the Upstream Cluster

We generally recommend running Rancher on a dedicated cluster, free of other workloads, to avoid potential performance and compatibility issues.

Rancher, especially when managing a growing number of clusters, nodes, and workloads, places a significant load on core Kubernetes components like `etcd` and `kube-apiserver` on the upstream cluster. Third-party software can interfere with the performance of these components and Rancher, potentially leading to instability.

Furthermore, third-party software can functionally interfere with Rancher. To minimize compatibility risks, deploy only essential Kubernetes system components and Rancher on the upstream cluster.

The following applications and components generally do not interfere with Rancher or the Kubernetes system:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs, CPIs, CSIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)
* the [Harbor](https://goharbor.io/) container registry

Note that each of these components has its own minimum resource requirements, which must be met in addition to Rancher's.

Container registries, in particular, can require significant bandwidth for serving images. Ensure sufficient bandwidth is available, ideally reserved using Quality of Service (QoS) mechanisms, for Rancher.

For high-scale deployments, consider dedicating separate nodes to non-Rancher software using [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to minimize interference.

The following software can interfere with Rancher performance at scale and is therefore discouraged on the upstream cluster:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))

### Make sure nodes are configured correctly for Kubernetes
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,23 +23,7 @@ When scaling up Rancher, one typical bottleneck is resource growth in the upstre

### Minimizing Third-Party Software on the Upstream Cluster

Running Rancher at scale can put significant load on internal Kubernetes components, such as `etcd` or `kubeapiserver`. Issues may arise if third-party software interferes with the performance of those components or with Rancher.

Every third-party piece of software carries a risk of interference. To prevent performance issues on the upstream cluster, you should avoid running any other apps or components, beyond Kubernetes system components and Rancher itself.

Software in the following categories generally won't interfere with Rancher or Kubernetes system performance:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)

On the other hand, the following software are found to interfere with Rancher performance at scale:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))
Recommendations outlined in the [general Rancher recommendations](./tips-for-running-rancher.md#minimize-third-party-software-on-the-upstream-cluster) are particularly important in a high scale context.

### Managing Your Object Counts

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This guide contains our recommendations for running the Rancher server, and is i

### Recommended Architecture and Infrastructure

Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server on a high-availability Kubernetes cluster.
Refer to this [guide](tips-for-running-rancher.md) for our general advice for setting up the Rancher server for a production installation.

### Deployment Strategies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,34 @@ If you are installing Rancher in a vSphere environment, refer to the best practi

When you set up your high-availability Rancher installation, consider the following:

### Run Rancher on a Separate Cluster
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
### Minimize Third-Party Software on the Upstream Cluster

We generally recommend running Rancher on a dedicated cluster, free of other workloads, to avoid potential performance and compatibility issues.

Rancher, especially when managing a growing number of clusters, nodes, and workloads, places a significant load on core Kubernetes components like `etcd` and `kube-apiserver` on the upstream cluster. Third-party software can interfere with the performance of these components and Rancher, potentially leading to instability.

Furthermore, third-party software can functionally interfere with Rancher. To minimize compatibility risks, deploy only essential Kubernetes system components and Rancher on the upstream cluster.

The following applications and components generally do not interfere with Rancher or the Kubernetes system:
* Rancher internal components, such as Fleet
* Rancher extensions
* Cluster API components
* CNIs, CPIs, CSIs
* Cloud controller managers
* Observability and monitoring tools (with the exception of prometheus-rancher-exporter)
* the [Harbor](https://goharbor.io/) container registry

Note that each of these components has its own minimum resource requirements, which must be met in addition to Rancher's.

Container registries, in particular, can require significant bandwidth for serving images. Ensure sufficient bandwidth is available, ideally reserved using Quality of Service (QoS) mechanisms, for Rancher.

For high-scale deployments, consider dedicating separate nodes to non-Rancher software using [taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to minimize interference.

The following software can interfere with Rancher performance at scale and is therefore discouraged on the upstream cluster:
* [CrossPlane](https://www.crossplane.io/)
* [Argo CD](https://argoproj.github.io/cd/)
* [Flux](https://fluxcd.io/)
* [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter) (see [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33))

### Make sure nodes are configured correctly for Kubernetes
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
Expand Down
Loading