feat: Add support to run notebooks on its own subdomains in Istio

[sven-ernw]

This PR is a draft at the moment, see Open Tasks for details.

This PR makes it possible to host notebooks on their own subdomains when Istio is used by adding this feature to the Kubeflow notebook-controller. This isolates the notebook's origin from the dashboard / Kubeflow API origin in the browser and addresses a security problem that allows session hijacking through a malicious notebook.

(reopend PR from !7742 in kubeflow/kubeflow)

Motivation / Why this change is needed

This change addresses a security problem that allows session hijacking through a malicious notebook:

An attacker can log session cookies by misusing the Notebook feature: An attacker can create a Notebook with a custom image that logs the cookies / trigger API requests. Afterwards, they must convince the victim to visit the URL of the Notebook (a phishing attack). This is a classic phishing attack scenario.

Disallowing custom images wouldn't be sufficient because you can achieve the same thing with a few extra steps using one of default images.

The problem relies on the fact that the Notebooks and the Kubeflow Dashboard / APIs are in the Browser's same origin. Thus, any attacker-controlled site would be able to access the Kubeflow API in the context of the victim.

To prevent this attack, the authenticating cookies need to be removed from requests forwarded to Notebook's Pods, and the notebooks need to be hosted on a different domain. This PR accomplishes this.

A proof of concept that demonstrates the problem is available but will not be disclosed until the problem is fixed.

Request for Feedback

At the moment, we are removing all cookie headers for security reasons before the request is forwarded to the notebook container. This breaks the functionality in some notebook images. So the goal is to remove only the authenticating cookie and keep the remaining cookies.

Our idea is to use a Istio Wasm Plugin that accomplishes this.

@kimwnasptd @juliusvonkohout @thesuperzapper @ederign What do you think?

What this PR accomplishes

1. Set individual hosts based on the environment variable ISTIO_HOST_NOTEBOOK in the notebook's VirtualServices when ISTIO_USE_NOTEBOOK_SUBDOMAINS is set to prevent access to the dashboard / APIs from attacker-controlled notebooks through the browser.
2. Implementing the authentication flow on the subdomains by redirecting unauthenticated requests to the host configured in ISTIO_HOST_AUTH for the ISTIO_AUTH_PATH.
3. Remove authenticating cookies from the request before it is forwarded to the notebook container (WIP) to prevent session token stealing.

Open Tasks

This PR is a draft at the moment because the following parts are work in progress:

1. At the moment, we are removing all cookie headers for security reasons before the request is forwarded to the notebook container. This breaks the functionality in some notebook images. So the goal is to remove only the authenticating cookie and keep the remaining cookies. See Request for Feedback
2. The subdomain feature requires additional configuration in the Kubeflow installation to make it fully functional (see configuration below). This will be documented and provided in a separate pull request in the manifests repository.
3. Unit test code for the new feature

Configuration

The subdomain feature is disabled by default as the setup has some prerequisites and is a breaking change. It requires a wildcard domain and wildcard TLS certificate or automated domain and certificate management.

To enable the feature, do the following configuration steps:

1. Configure the following environment variables in the notebook-controller:

ISTIO_USE_NOTEBOOK_SUBDOMAINS: true
ISTIO_HOST: "${NAMESPACE}-notebook-${NAME}.kubeflow.example.com"
ISTIO_HOST_NOTEBOOK: "${NAMESPACE}-notebook.kubeflow.example.com"
ISTIO_HOST_AUTH: "kubeflow.example.com"
ISTIO_HOST_AUTH_PATH: "/oauth2/"

2. The configmap of oauth2 must also be adjusted by enabling cookies for subdomains. Edit the oauth2_proxy.cfg configuration of the oauth2-proxy-hk55gm96k4 ConfigMap in the oauth2-proxy namespace and add the following setting:

cookie_domains = "kubeflow.example.com"

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thesuperzapper for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• components/notebook-controller/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sven-ernw]

Hi @kimwnasptd @juliusvonkohout @thesuperzapper @ederign,
we can join the Notebooks meeting next Thursday to discuss the reason for this feature and request for feedback part if you like.

CC: @lorinl

[andyatmiami]

/area v1