Add a Claude generated OVERVIEW.md file#17
Conversation
Signed-off-by: Nick Lathe <nick.lathe@code.org>
| **Key UIs:** | ||
|
|
||
| - ArgoCD: https://argocd.k8s.code.org | ||
| - Kargo: https://kargo.k8s.code.org |
There was a problem hiding this comment.
I encountered an error on google.com attempting to Single Sign On here.
| | Deployment | Environment | RAILS_ENV | DNS | Notes | | ||
| |---|---|---|---|---| | ||
| | **staging** | staging | staging | studio.staging.k8s.code.org | Auto-promoted by Kargo | | ||
| | **test** | test | staging | — | Manual promote from staging | |
There was a problem hiding this comment.
why does test env use staging rails env?
There was a problem hiding this comment.
Looks like Claude hallucinated this. Test env uses test rails env, not staging https://github.com/code-dot-org/k8s-gitops/blob/main/apps/codeai/envTypes/test/deployment.patch.yaml#L11-L12
There was a problem hiding this comment.
Think that might be an error - the actual env has RAILS_ENV set to test: https://github.com/code-dot-org/k8s-gitops/blob/main/apps/codeai/envTypes/test.values.yaml
|
|
||
| Secrets follow a consistent pattern: | ||
|
|
||
| 1. **Bootstrap**: secrets are created in AWS Secrets Manager by the OpenTofu `cluster-infra` module, prefixed with `k8s/tofu/<cluster>/` |
There was a problem hiding this comment.
Are secrets for an application defined in this repo or in the application's repo?
There was a problem hiding this comment.
Secrets are handled by the External Secrets Operator, which pulls in secrets from AWS secrets manager. The secrets referenced here are the ones necessary for actually bootstrapping the cluster.
| - **mimic/** — A small replica of the real `apps/` tree structure. Use it to test ArgoCD behavior (app-of-apps recursion, ApplicationSet wrappers, Ingress) without affecting production. All resource names start with `mimic-`. | ||
| - **argo-trace-stress-test/** — A broader stress harness for `argo-trace` covering edge cases: broken workloads, finalizer stalls, Crossplane resources, hook Jobs. | ||
|
|
||
| When modifying `apps/app-of-apps/*`, make a parallel edit in `bootstrap/apptrees/mimic/apps/app-of-apps/*` to prevent bitrot. |
There was a problem hiding this comment.
Is there a way to automatically enforce this? Like a commit/ci hook that errors if there is an edit of a file in one directory but not the other?
There was a problem hiding this comment.
Good question - I'm not sure how much we will need mimic long-term
| | **staging** | staging | staging | studio.staging.k8s.code.org | Auto-promoted by Kargo | | ||
| | **test** | test | staging | — | Manual promote from staging | |
There was a problem hiding this comment.
From a security perspective, I prefer non-production systems to use a different base domain name:
staging --> k8s.staging-code.org
test --> k8s.test-code.org
|
high level thought ... I'm not sure we're ready to adopt this new infrastructure framework before the school year starts in 3-4 weeks. Could we start out with Solid Queue running as an ECS Service while we build out this infrastructure? |
| | **staging** | staging | staging | studio.staging.k8s.code.org | Auto-promoted by Kargo | | ||
| | **test** | test | staging | — | Manual promote from staging | | ||
| | **production** | production | production | studio.production.k8s.code.org | Manual promote from test | |
There was a problem hiding this comment.
Opportunity to clean up the TLD?
| Values are layered (later overrides earlier): | ||
|
|
||
| 1. **Base chart** — from `code-dot-org/code-dot-org` repo at `k8s/helm` | ||
| 2. **Environment type** — `apps/codeai/envTypes/<envType>.values.yaml` (sets `RAILS_ENV`, health checks, scheduling, autoscaling defaults) |
There was a problem hiding this comment.
Opportunity to rename test? (Although we don't need to if we don't want to couple it)
There was a problem hiding this comment.
will migrating to k8s make it any easier to change the name later?
There was a problem hiding this comment.
Possibly, we can do it later too as long as we design it so that an environment can be rebuilt from scratch.
|
|
||
| Kargo handles progressive delivery of new container images through environments. | ||
|
|
||
| ### How it works |
There was a problem hiding this comment.
It is unclear how images are built and how it flows into Argo. Depending on the direction, we will probably need a patcher which adds a "private" layer to the open source image for security hot fixes.
| cd bootstrap/codeai-k8s/cluster | ||
| tofu init && AWS_PROFILE=codeorg-admin tofu apply | ||
| ``` | ||
| Creates the EKS Auto Mode cluster, VPC, subnets, NAT gateways, security groups, KMS key, and OIDC provider. |
There was a problem hiding this comment.
What networking provider are we choosing here? AWS defaults? Or are we interested in something like Cillium and establishing defense-in-depth? Might be worth looking into if we're starting a new cluster.
That may be a way to address @cat5inthecradle concern of "do we want all k8's application environments in the same cluster" - we can namespace the environments and Cilllium can firewall them (if we choose a single cluster). Caveat being that AWS may provide the same functionality that Cillium does, but I do know optionality may be something to look at
|
|
||
| ### Log files | ||
|
|
||
| When `log-cluster-events` is running, it writes three logs: |
There was a problem hiding this comment.
Send to Cloudwatch instead perhaps?
| Automatic — Kargo promotes new images from the warehouse to staging as soon as they appear. | ||
|
|
||
| ### Promote an image from staging to test (or test to production) | ||
| Use the [Kargo UI](https://kargo.k8s.code.org) to manually promote freight between stages. |
There was a problem hiding this comment.
What would it take to automate? UI Tests?
| 3. Merge to `main` | ||
|
|
||
| ### Bump an app Helm chart | ||
| When modifying an app's Helm chart under `apps/`, always bump the `version:` field in its `Chart.yaml` so ArgoCD detects the change. |
There was a problem hiding this comment.
Could we automate this or make it reconcile based on an image being published on GHCR?
| ``` | ||
| Creates secrets in AWS Secrets Manager, IAM roles, and generates `apps/infra/codeai-cluster-config.values.yaml` (cluster facts consumed by Helm charts). | ||
|
|
||
| First-time bootstrap requires setting `dex_google_client_secret`, `kargo_k8s_gitops_repo_username`, and `kargo_k8s_gitops_repo_password` in `terraform.tfvars`. Remove secrets from tfvars after the first apply. |
There was a problem hiding this comment.
The reliance on "remove secrets from tfvars after the first apply" may unintentionally result in secrets leakage in PR.
|
Do we need to reproduce the |
7 participants
No description provided.