Skip to content

Commit

Permalink
Control API Architecture (#63)
Browse files Browse the repository at this point in the history
* add instance definitions to glossary

* control api architecture
tobru authored Nov 9, 2021
1 parent f3feda1 commit 89e0ca4
Showing 7 changed files with 350 additions and 0 deletions.
127 changes: 127 additions & 0 deletions docs/modules/ROOT/pages/references/architecture/control-api-org.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
= {controlapi}: Organization

TIP: This resource implements the xref:references/functional-requirements/portal.adoc#_feature_manage_organizations["Manage Organizations"] features.

List operations on Kubernetes resources can not be filtered by RBAC, it's a binary operation:
Resources can either be listed or not, there is no way to give access only to a sub-set.
To circumvent this limitation, `Organization` is a virtual resource.

The `Organization` resource represents a filtered and formatted list of standard Kubernetes `Namespace` resources which have specific labels and annotations.

It is assumed that the `Organization` resource is used for all operations, the represented `Namespace` must not be directly manipulated.

== Object

.Virtual resource
[source,yaml]
----
apiVersion: appuio.io/v1
kind: Organization
metadata:
name: acme-corp <1>
annotations:
organization.appuio.io/namespace: org-acme-corp <2>
spec:
displayName: Acme Corp. <3>
----
Field mapping from the represented `Namespace` resource:

<1> `metadata.labels[appuio.io/metadata.name]`
<2> `metadata.name`
<3> `metadata.annotations[organization.appuio.io/display-name]`

An https://book.kubebuilder.io/reference/generating-crd.html#additional-printer-columns[additional printer column] can help to identify the associated namespace resource name.
This is useful when working with organization-scoped objects which are available in the organization's namespace.

.Original resource
[source,yaml]
----
apiVersion: v1
kind: Namespace
metadata:
name: org-acme-corp <1>
labels:
appuio.io/resource.type: organization <2>
appuio.io/metadata.name: acme-corp <3>
annotations:
organization.appuio.io/display-name: Acme Corp. <4>
----
<1> Resource name, prefixed with `org-` to circumvent possible name collision
<2> Identify resource type, used by the API server to filter for namespaces representing organizations
<3> `metadata.name` of the virtual `Organization` object
<4> Reflected in the `Organization` object as `spec.displayName`

== Labels and Annotations

[cols="2,1,1,3",options="header"]
|===
|Name
|Type
|Resource
|Description

|`appuio.io/resource.type`
|label
|`v1/Namespace`
|Identifies the resource type in the scope of the {controlapi}

|`appuio.io/metadata.name`
|label
|`v1/Namespace`
|`metadata.name` of the virtual resource

|`organization.appuio.io/display-name`
|annotation
|`appuio.io/v1/Organization`
|Display name of the organization

|===

== Resource filter

The virtual resource is a filtered view of `Namespaces`.
The filter uses the following heuristic:

* API version: `v1`
* Kind: `Namespace`
* Label: `appuio.io/resource-type=organization`
* Subject is bound to one of the defined `ClusterRole` resources.

== RBAC and Cluster roles

These are `ClusterRole` resources which are bound to a subject by a namespaced `RoleBinding`:

`org-view`:: View (read only) access to an organization
`org-admin`:: Admin (read / write) access to an organization

By default, creating organizations can be done by all authenticated users.

== Organization Membership

All members of an organization are configured in an `OrganizationMembers` resource.

.CRD based
[source,yaml]
----
apiVersion: appuio.io/v1
kind: OrganizationMembers
metadata:
name: acme-corp-members
namespace: org-acme-corp
spec:
userRefs: <1>
- id: bec0d928-2ae2-4cec-94a0-5f72f12b8b39
- username: peter.muster
status:
resolvedUserRefs: <2>
- id: bec0d928-2ae2-4cec-94a0-5f72f12b8b39
username: kate.demo
- id: 508a9160-977c-4c57-963f-c7b511c4ecc5
username: peter.muster
----
<1> References to one or more xref:references/architecture/control-api-user.adoc[`User`] resource. +
Only one of the two parameters are allowed:

* `id` must match `metadata.name` of an existing `User` resource
* `username` must match `spec.username` from an existing `User` resource
<2> This is resolved by the xref:explanation/system/details-adapters.adoc[adapter]
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
= {controlapi}: Team

TIP: This resource implements the xref:references/functional-requirements/portal.adoc#_feature_manage_teams["Manage Teams"] features.

== Object

.CRD based
[source,yaml]
----
apiVersion: appuio.io/v1
kind: Team
metadata:
name: myteam1
namespace: org-acme-corp <1>
spec:
displayName: My Super Team 1
userRefs: <2>
- id: bec0d928-2ae2-4cec-94a0-5f72f12b8b39
- username: peter.muster
status:
resolvedUserRefs: <3>
- id: bec0d928-2ae2-4cec-94a0-5f72f12b8b39
username: kate.demo
- id: 508a9160-977c-4c57-963f-c7b511c4ecc5
username: peter.muster
----
<1> The organizations namespace
<2> References to one or more xref:references/architecture/control-api-user.adoc[`User`] resource. +
Only one of the two parameters are allowed:

* `id` must match `metadata.name` of an existing `User` resource
* `username` must match `spec.username` from an existing `User` resource
<3> This resolved by the xref:explanation/system/details-adapters.adoc[adapter]
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
= {controlapi}: User

TIP: This resource implements the xref:references/functional-requirements/portal.adoc#_feature_personal_settings_per_user["Personal Settings per User"] feature.

== Object

.CRD based
[source,yaml]
----
apiVersion: appuio.io/v1
kind: User
metadata:
name: bec0d928-2ae2-4cec-94a0-5f72f12b8b39 <1>
spec:
displayName: Kate Demo
email: [email protected]
defaultOrganizationRef: acme-corp
status: <2>
displayName: Kate Demo
username: kate.demo
email: [email protected]
defaultOrganizationRef: acme-corp
----
<1> User ID in Keycloak
<2> Reflects actual configuration from adapter and exposes read-only fields

== Access control

As this is a cluster-scoped resource, access control has to be defined in the cluster scope.
For each `User` object, a `ClusterRole` and `ClusterRoleBinding` is generated in the background, which grants edit rights to the subject (owner) which relates to the `User`.

[source,yaml]
----
kind: ClusterRole
metadata:
name: bec0d928-2ae2-4cec-94a0-5f72f12b8b39-owner
rules:
- apiGroups: ["appuio.io"]
resources: ["users"]
resourceNames: ["bec0d928-2ae2-4cec-94a0-5f72f12b8b39"]
verbs: ["get", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: bec0d928-2ae2-4cec-94a0-5f72f12b8b39-owner
subjects:
- kind: User
name: appuio#bec0d928-2ae2-4cec-94a0-5f72f12b8b39 <1>
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: bec0d928-2ae2-4cec-94a0-5f72f12b8b39-owner
apiGroup: rbac.authorization.k8s.io
----
<1> This depends on the API server configuration: +
`oidc-username-claim=sub` and `oidc-username-prefix=appuio#`

By default, only the subject which is the owner of the `User` object gets edit rights.
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
= {controlapi}: Zone

TIP: This resource implements the xref:references/functional-requirements/portal.adoc#_feature_zones["Manage Zones"] features.

== Object

.CRD based
[source,yaml]
----
apiVersion: appuio.io/v1
kind: Zone
metadata:
name: cloudscale-lpg-0
data:
displayName: cloudscale.ch LPG 0
features:
openshift-version: "4.8"
kubernetes-version: "1.21"
sdn: OVN-Kubernetes
urls:
console: https://console.cloudscale-lpg-0.appuio.cloud/
kubernetes-api: https://api.cloudscale-lpg-0.appuio.cloud:6443/
registry: https://registry.cloudscale-lpg-0.appuio.cloud
logging: https://logging.cloudscale-lpg-0.appuio.cloud/
cname: cname.cloudscale-lpg-0.appuio.cloud
default-app-domain: apps.cloudscale-lpg-0.appuio.cloud
gateway-ips:
- 185.98.123.122
cloud-provider:
name: cloudscale.ch
zones:
- lpg1
region: Lupfig (AG)
----
82 changes: 82 additions & 0 deletions docs/modules/ROOT/pages/references/architecture/control-api.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
= {controlapi} Architecture

This page describes the xref:references/glossary.adoc#_control_api[{controlapi}].
It adheres to the decision taken in xref:explanation/decisions/control-api.adoc[{controlapi}] and it follows the features asked in xref:references/functional-requirements/portal.adoc[Functional Requirements Targeting the Portal].

== General Principles

Kubernetes API::
The Control API is built upon the Kubernetes API and adheres to it's https://kubernetes.io/docs/reference/kubernetes-api/[design principles].

API object field: `.status`::
The `status` field (also called https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#status-subresource["Status subresource"]) represents the current observed state and external information.
It is read-only to the end-user.

API object field: `.spec`::
The `spec` field contains the desired state for reconciliation.

== Virtual vs. CRD based resources

Some resources are not persisted to etcd and are only available _virtually_, others are persisted to etcd and are defined and represented via `CustomResourceDefinition` (CRDs).

Virtual resources are accessible via an https://kubernetes.io/docs/tasks/extend-kubernetes/setup-extension-api-server/[API Server Extension].
These resources are similar to views in a relational database.
The benefit of providing these resources instead of only using CRDs is that we can calculate access permissions dynamically for every request.
The same concept is also used by OpenShift with its `Project` resource which represents RBAC filtered `Namespaces` (see https://github.com/openshift/kube-projects[kube-projects]).
And we can also find it in https://github.com/loft-sh/kiosk[Kiosk] for example.

== Authentication and Authorization

Authentication against the API server is done by the {idp}.
It's always the same subject (user) which is being used throughout the whole {product} ecosystem.

For authorization, standard https://kubernetes.io/docs/reference/access-authn-authz/rbac/[Kubernetes RBAC] is being used.
Kyverno policies can be used to implement enhanced policies, for example the number of resources of a specified kind a user is allowed to create.

There are several layers of authorization:

* Kubernetes RBAC
* Kyverno policies
* Virtual resources with filtering

== Resource Scopes

Resources on the Kubernetes API server can either be https://kubernetes.io/docs/reference/using-api/api-concepts/#standard-api-terminology[cluster scoped or namespace scoped].

Each {controlapi} instance is represented by one Kubernetes API server instance.
This allows us to leverage the scoping concept of the Kubernetes API server to reflect the scopes in the {product} domain.
Also, by doing it that way, standard Kubernetes RBAC rules can be used for permission handling on an organization level.

{global} resources are available on the Kubernetes global scope (no namespace) whereas organization level resources are namespace scoped.

Global:: Kubernetes cluster global resource.
Organization:: Kubernetes namespaced resource.

== Resources

[cols="1,1,2",options="header"]
|===
|Name
|Scope
|Type

|xref:references/architecture/control-api-org.adoc[Organization]
|Global
|Virtual (represents filtered `Namespace` resources)

|xref:references/architecture/control-api-org.adoc#_organization_membership[OrganizationMembers]
|Organization
|CRD

|xref:references/architecture/control-api-user.adoc[User]
|Global
|CRD

|xref:references/architecture/control-api-team.adoc[Team]
|Organization
|CRD

|xref:references/architecture/control-api-zone.adoc[Zone]
|Global
|CRD
|===
10 changes: 10 additions & 0 deletions docs/modules/ROOT/pages/references/glossary.adoc
Original file line number Diff line number Diff line change
@@ -19,6 +19,9 @@ A Kubernetes namespace (in OpenShift also an object type "Project").

== {product} specific

=== {product} Instance
An instance of an {product} installation, consisting of a {global} instance and a number of Zones.

=== Zone
A single OpenShift cluster that is part of an {product} offering.

@@ -60,6 +63,13 @@ A user which interacts with {product} (the Kubernetes API) in a specified namesp
=== Organization Owner
A user which interacts with {product} (the Kubernetes API).

=== Instance Owner
Owner of the particular {product} instance.

=== Instance Admin
Administrator of the particular {product} instance.
On a {zone} it's the cluster admin, on the global level it's the admin.

== Roles

OpenShift default roles are documented in the https://docs.openshift.com/container-platform/4.9/authentication/using-rbac.html#default-roles_using-rbac[upstream documentation].
5 changes: 5 additions & 0 deletions docs/modules/ROOT/partials/nav-reference.adoc
Original file line number Diff line number Diff line change
@@ -11,6 +11,11 @@
** xref:appuio-cloud:ROOT:references/architecture/namespace-ownership.adoc[Namespace ownership]
** xref:appuio-cloud:ROOT:references/architecture/metering.adoc[Metering of resource usage]
** xref:appuio-cloud:ROOT:references/architecture/metrics-of-interest.adoc[Metrics of interest]
** xref:appuio-cloud:ROOT:references/architecture/control-api.adoc[Control API]
*** xref:appuio-cloud:ROOT:references/architecture/control-api-org.adoc[Organization]
*** xref:appuio-cloud:ROOT:references/architecture/control-api-user.adoc[User]
*** xref:appuio-cloud:ROOT:references/architecture/control-api-team.adoc[Team]
*** xref:appuio-cloud:ROOT:references/architecture/control-api-zone.adoc[Zone]

* Functional Requirements

0 comments on commit 89e0ca4

Please sign in to comment.