Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support certificate rotation #206

Closed
jbeda opened this issue Mar 17, 2017 · 17 comments · Fixed by kubernetes/kubernetes#67910
Closed

Support certificate rotation #206

jbeda opened this issue Mar 17, 2017 · 17 comments · Fixed by kubernetes/kubernetes#67910
Assignees
@stealthybox @liztio
Labels
area/security kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triaged
Milestone
v1.12

Comments

@jbeda
Copy link

jbeda commented Mar 17, 2017

[FEATURE REQUEST]

We need some way to rotate certificates. Right now we mint certs that expire after 1 year and we have no built in provision to rotate them.

We should either have components manage rotating their own certificates (and writing the certs back to the kubeconfig) or have a way to easily do this from the outside without disrupting a running cluster. Notifying users when rotation is necessary should be part of our design here too.
@jbeda jbeda mentioned this issue Mar 17, 2017
Merged
@pipejakob
Copy link
Contributor

See also kubernetes/kubernetes#4672

@jimmycuadra
Copy link

See also https://github.com/kubernetes/kubernetes/issues/25379

@luxas luxas added area/security kind/tracking-issue priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels May 29, 2017
@luxas
Copy link
Member

luxas commented May 29, 2017

@timothysc @pipejakob @mikedanese @jcbsmpsn @jbeda
Should we make this a goal for v1.8 with self-hosting and rotateable kubelet certs?

@jcbsmpsn
Copy link

jcbsmpsn commented May 30, 2017 •
edited
Loading

I think there are still a few steps left to accomplishing this:

Remaining:

  • When the kubelet boots on a master, and doesn't point to an API server, it has a self signed cert that isn't rotated.
  • The API server has a signed certificate that needs to be rotated.
  • The CA for the cluster should be rotated/rotatable too.
  • Key/Cert rotation for Kubernetes clients. kubernetes#4672 covers offering the rotation logic to all clients
  • Offering a mechanism to do this on demand seems like a useful perk (my admin just left the company and I'd like to rotate all cert/keys in the cluster)

All these might be a stretch for 1.8.

@jcbsmpsn
Copy link

/assign @jcbsmpsn

@luxas luxas added this to the v1.8 milestone Jun 19, 2017
@luxas
Copy link
Member

luxas commented Sep 7, 2017

Sub-goal of v1.8 is getting kubelet client cert rotation enabled, ref #386
Other than that, I'm moving this out for further improvements in v1.9 as new auth features regarding this are available.

@luxas luxas modified the milestones: v1.9, v1.8 Sep 7, 2017
@nathanleclaire
Copy link

It would be cool if somehow this could be configured to run automatically, via cron or something like that, like Docker Swarm mode does. Swarm mode is constantly refreshing certificates, which, to be fair, has its own set of tradeoffs (example: if a worker node panics/partitions and stops refreshing certs for a while, it's locked out of the cluster. not sure if it still happens, but that was one issue we ran into when the feature was new)

@jcbsmpsn
Copy link

jcbsmpsn commented Oct 3, 2017

@nathanleclaire Not sure what you mean. When certificate rotation is enabled it does rotate the certificates automatically. Currently only the certificate used by the node is done. More work to do for other certificates used by Kubernetes.

@nathanleclaire
Copy link

@jcbsmpsn Ah, cool. Still catching up on all the fast-moving new Kube stuff :)

@jamiehannaford
Copy link
Contributor

Now that kubelet supports auto-rotation, should we enable this feature gate in 1.9? Would we also need to feature gate it? If so, maybe we can make it generic (e.g. AllowAutoTLSRotation) to pave the way for other components in the future.

@luxas
Copy link
Member

luxas commented Oct 20, 2017

We already enable kubelet client cert autorotation. We'll enable kubelet servingcert rotation if possible in v1.9

@luxas luxas self-assigned this Oct 20, 2017
@luxas
Copy link
Member

luxas commented Nov 18, 2017

We'll enable kubelet servingcert rotation if possible in v1.9

Not possible yet AFAIK.

In the interim we'll do #548

@luxas luxas modified the milestones: v1.9, v1.10 Nov 18, 2017
@xiangpengzhao xiangpengzhao mentioned this issue Nov 21, 2017
Merged
2 tasks
@errordeveloper errordeveloper mentioned this issue Jan 22, 2018
Closed
@timothysc timothysc assigned stealthybox and unassigned luxas and jcbsmpsn Jan 24, 2018
@timothysc
Copy link
Member

Ideally we could have a phase command to help rotate.
/cc @mattmoyer and @mikedanese (re: other googlers)

@stealthybox
Copy link
Member

Suggested MVP is that you would run the phase certs rotate command per node.
@timothysc suggests we have a daemonset with hostpath mounts assist in this rotation/upgrade.

@errordeveloper errordeveloper mentioned this issue Jan 26, 2018
Closed
@mattkelly mattkelly mentioned this issue Feb 21, 2018
Closed
@timothysc timothysc modified the milestones: v1.10, v1.11 Mar 5, 2018
@SpComb
Copy link

SpComb commented Mar 20, 2018

Related to kubelet server certs, once the kubelet uses a kube-issued server cert instead of the current self-signed server cert, then the kube-apiserver also needs --kubelet-certificate-authority to validate those kubelet server certs.

@luxas luxas modified the milestones: v1.11, v1.12 May 22, 2018
@timothysc
Copy link
Member

/assign @liztio

@timothysc timothysc added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/feature Categorizes issue or PR as related to a new feature. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. kind/tracking-issue labels Jul 3, 2018
@liztio liztio added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Jul 24, 2018
@liztio
Copy link

liztio commented Jul 25, 2018

I've done some testing on the client cert upgrades. My methodology:

  1. Join a node to the cluster using kubeadm join
  2. make a csr for the existing private key with openssl req
  3. Sign the csr on the master using the k8s ca, but set it to expire in two days (-days 2)
  4. copy the issued cert back, replace it on the worker node
  5. upgrade the node (in my case, 1.11.0 -> 1.11.1).

Based on my reading of the merged features, I should've seen a new certificate for the client node be issued, but so far that hasn't happened. I'll look into this in more detail tomorrow.

@liztio liztio mentioned this issue Aug 2, 2018
Closed
@neolit123 neolit123 mentioned this issue Aug 14, 2018
Closed
@liztio liztio mentioned this issue Aug 27, 2018
Merged
@timothysc timothysc mentioned this issue Aug 30, 2018
Closed
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Aug 31, 2018
Merge pull request #67910 from liztio/cert-renewal
17dde46 
Automatic merge from submit-queue (batch tested with PRs 64283, 67910, 67803, 68100). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.

Kubeadm Cert Renewal

**What this PR does / why we need it**:

adds explicit support for renewal of certificates via command

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes kubernetes/kubeadm#206

**Special notes for your reviewer**:
The targeted documentation is at kubernetes/website#9712

**Release note**:

```release-note
Adds the commands `kubeadm alpha phases renew <cert-name>`
```
k8s-publishing-bot added a commit to kubernetes/client-go that referenced this issue Sep 1, 2018
@k8s-publishing-bot
Merge pull request #67910 from liztio/cert-renewal
ec724c2 
Automatic merge from submit-queue (batch tested with PRs 64283, 67910, 67803, 68100). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.

Kubeadm Cert Renewal

**What this PR does / why we need it**:

adds explicit support for renewal of certificates via command

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes kubernetes/kubeadm#206

**Special notes for your reviewer**:
The targeted documentation is at kubernetes/website#9712

**Release note**:

```release-note
Adds the commands `kubeadm alpha phases renew <cert-name>`
```

Kubernetes-commit: 17dde46baebe0b67421132af7b99b42d89ea4cd0
sttts pushed a commit to sttts/client-go that referenced this issue Sep 5, 2018
@k8s-publishing-bot
Merge pull request #67910 from liztio/cert-renewal
2c1924d 
Automatic merge from submit-queue (batch tested with PRs 64283, 67910, 67803, 68100). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.

Kubeadm Cert Renewal

**What this PR does / why we need it**:

adds explicit support for renewal of certificates via command

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes kubernetes/kubeadm#206

**Special notes for your reviewer**:
The targeted documentation is at kubernetes/website#9712

**Release note**:

```release-note
Adds the commands `kubeadm alpha phases renew <cert-name>`
```

Kubernetes-commit: 17dde46baebe0b67421132af7b99b42d89ea4cd0
k8s-publishing-bot added a commit to kubernetes/client-go that referenced this issue Sep 6, 2018
@k8s-publishing-bot
Merge pull request #67910 from liztio/cert-renewal
cb4babb 
Automatic merge from submit-queue (batch tested with PRs 64283, 67910, 67803, 68100). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.

Kubeadm Cert Renewal

**What this PR does / why we need it**:

adds explicit support for renewal of certificates via command

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes kubernetes/kubeadm#206

**Special notes for your reviewer**:
The targeted documentation is at kubernetes/website#9712

**Release note**:

```release-note
Adds the commands `kubeadm alpha phases renew <cert-name>`
```

Kubernetes-commit: 17dde46baebe0b67421132af7b99b42d89ea4cd0
@liztio liztio mentioned this issue Sep 19, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stealthybox stealthybox

@liztio liztio

Labels
area/security kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triaged
Projects
None yet
Milestone
v1.12