Skip to content

Linkerd Tap doesn't seem to work with EKS Access Entries authentication #13169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
multimac opened this issue Oct 11, 2024 · 2 comments · May be fixed by #13170
Open

Linkerd Tap doesn't seem to work with EKS Access Entries authentication #13169

multimac opened this issue Oct 11, 2024 · 2 comments · May be fixed by #13170
Labels
bug env/eks Amazon EKS

Comments

@multimac
Copy link
Contributor

multimac commented Oct 11, 2024

What is the issue?

I was trying to use linkerd viz tap to debug an internal issue, but kept getting tap authorization failed. After looking at the documentation page in the error message (https://linkerd.io/2.16/tasks/securing-linkerd-tap/), it seemed like the problem might be in Linkerd itself as kubectl can-i works

How can it be reproduced?

I believe this might need an EKS cluster configured with EKS Access Entries for authentication :/

Logs, error output, etc

$ kubectl auth can-i watch deployments.v1alpha1.tap.linkerd.io/[xxx] --namespace [xxx] --subresource tap
yes
$ linkerd viz tap deployment/[xxx] --namespace [xxx]
HTTP error, status Code [403] (unexpected API response: {"error":"tap authorization failed (not authorized to access deployments.tap.linkerd.io), visit https://linkerd.io/tap-rbac for more information"})

output of linkerd check -o short

linkerd-identity
----------------
× trust anchors are using supported crypto algorithm
    Invalid trustAnchors:
        * 498364598153846236906649535816328601770266843336 One Model Linkerd Root CA must use P-256 curve for public key, instead P-384 was used
    see https://linkerd.io/2/checks/#l5d-identity-trustAnchors-use-supported-crypto for hints

linkerd-jaeger
--------------
‼ jaeger extension proxies are up-to-date
    some proxies are not running the current version:
        * collector-9cb9f47cb-2p4dn (edge-24.10.1)
        * collector-9cb9f47cb-s8zg2 (edge-24.10.1)
        * jaeger-injector-759b7ff6fb-5xcmz (edge-24.10.1)
        * jaeger-injector-759b7ff6fb-q46tg (edge-24.10.1)
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cp-version for hints
‼ jaeger extension proxies and cli versions match
    collector-9cb9f47cb-2p4dn running edge-24.10.1 but cli running edge-24.10.2
    see https://linkerd.io/2/checks/#l5d-jaeger-proxy-cli-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * alertmanager-alertmanager-0 (edge-24.10.1)
        * metrics-api-fd554c577-8htqz (edge-24.10.1)
        * metrics-api-fd554c577-ftj2d (edge-24.10.1)
        * prometheus-prometheus-0 (edge-24.10.1)
        * tap-5579f7499d-2v4dg (edge-24.10.1)
        * tap-5579f7499d-5jf5j (edge-24.10.1)
        * tap-5579f7499d-bmdfg (edge-24.10.1)
        * tap-injector-5bfbc5b658-fqr7c (edge-24.10.1)
        * tap-injector-5bfbc5b658-kvz6s (edge-24.10.1)
        * tap-injector-5bfbc5b658-mrlmg (edge-24.10.1)
        * web-694c5b64d8-hs76l (edge-24.10.1)
        * web-694c5b64d8-kkqtp (edge-24.10.1)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    alertmanager-alertmanager-0 running edge-24.10.1 but cli running edge-24.10.2
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints

Status check results are ×

Environment

Kubernetes Version: v1.30
Cluster Environment: AWS EKS
Host OS: Amazon Bottlerocket
Linkerd version: Client version: edge-24.10.2 / Server version: edge-24.10.1

Possible solution

I've made a commit which I will push up in a PR shortly, but I suspect the issue may be because the SubjectAccessReview done by the Tap controller doesn't pass in any of the "extra" user attributes

When looking at the audit logs generated by our Kubernetes control plane and comparing the linkerd viz tap vs. kubectl can-i, I can see that kubectl can-i is passing some additional "extra" fields that seem relevant to the EKS Access Entries authentication.

My commit updates ResourceAuthzForUser in linkerd/pkg/k8s/authz.go to take in the list of extra attributes, retrieved via the X-Remote-Extras- HTTP header

Additional context

No response

Would you like to work on fixing this bug?

yes

@multimac multimac added the bug label Oct 11, 2024
@olix0r olix0r added the env/eks Amazon EKS label Oct 11, 2024
multimac added a commit to multimac/linkerd2 that referenced this issue Oct 11, 2024
Kubernetes authorization plugins can rely on extra attributes on a user, and these are provided via `X-Remote-Extra-` headers. Currently the Linkerd Viz `tap` API doesn't include these attributes when making the `SubjectAccessReview` request which means the Tap API cannot be used by end-users who's clusters use such authz plugins.

This change updates the `tap` controller to parse the `X-Remote-Extra-` headers and include them in the SubjectAccessReview request.
Fixed linkerd#13169

Signed-off-by: David Symons <[email protected]>
@multimac multimac linked a pull request Oct 11, 2024 that will close this issue
Copy link

stale bot commented Jan 9, 2025

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jan 9, 2025
@wmorgan wmorgan removed the wontfix label Jan 9, 2025
@jseiser
Copy link
Contributor

jseiser commented Mar 19, 2025

We have this same issue.

Kubernetes Version: v1.31
Cluster Environment: AWS EKS
Host OS: Amazon Bottlerocket
linkerd version
Client version: edge-25.3.2
Server version: edge-25.1.2

We can not tap in the UI, or on the CLI

> kubectl auth can-i watch pods.tap.linkerd.io --all-namespaces --as system:serviceaccount:linkerd-viz:web
yes
 linkerd viz tap -n linkerd deploy/linkerd-controller --as $(whoami)
Cannot connect to Linkerd Viz: namespaces is forbidden: User "justin" cannot list resource "namespaces" in API group "" at the cluster scope
Validate the install with: linkerd viz check

check's are fine

> linkerd viz check
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ can initialize the client
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * metrics-api-55c95c4f49-4ds2t (edge-25.1.2)
        * metrics-api-55c95c4f49-g44p2 (edge-25.1.2)
        * prometheus-6d4d66c58d-585m6 (edge-25.1.2)
        * tap-7b958b9784-qf5cb (edge-25.1.2)
        * tap-7b958b9784-skknm (edge-25.1.2)
        * tap-injector-769865c97-d4ngv (edge-25.1.2)
        * tap-injector-769865c97-r2srb (edge-25.1.2)
        * web-6cfd8f5bb7-9rg47 (edge-25.1.2)
        * web-6cfd8f5bb7-lh58f (edge-25.1.2)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-55c95c4f49-4ds2t running edge-25.1.2 but cli running edge-25.3.2
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
√ prometheus is installed and configured correctly
√ viz extension self-check

Status check results are √

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug env/eks Amazon EKS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants