DC-1227 - Reduce costs by shutting down personal deployments when not in use #1822
Conversation
snf2ye
requested review from
fboulnois and
rushtong
and removed request for
a team
docs/jade-getting-started.md
Outdated
| However, there are still some use cases for personal dev environments. So, you should feel free to | ||
| use one of the existing personal dev environments for your work (but check with the data repo team to | ||
| see if anyone else is using a particular environment). |
There was a problem hiding this comment.
Note to reviewers: The terraform code to create personal deployments is in the terraform-jade repository (that is now archived). My opinion is that we can just use existing namespace configurations - it doesn't really matter whose name was originally assigned to them. We shouldn't set up new personal deployments for new members of the team.
pshapiro4broad
left a comment
pshapiro4broad
left a comment
There was a problem hiding this comment.
Looks OK. Not directly related to your change, but I believe the invocation of db-connect.sh in this doc is no longer valid.
docs/jade-getting-started.md
Outdated
| cd jade-data-repo/ops | ||
| DB=datarepo-ZZ SUFFIX=ZZ ENVIRONMENT=dev ./db-connect.sh |
There was a problem hiding this comment.
It looks like this wasn't updated as part of #1731
Actually looking at this more closely I'm not sure how to invoke the script for this case. @fboulnois can you provide the new command line for this operation?
There was a problem hiding this comment.
That functionality is no longer supported in the db-connect.sh script. This decision was based on feedback as a part of #1731 (comment)
There was a problem hiding this comment.
After looking more into this, given that personal deployments will mainly be used for testing helm charts, I don't expect for there to be extensive postgres querying. Users can access the database in the Gcloud UI for some one-off queries.
|
Also I see that this PR causes test to run. Since it's a documentation only change, can we update our test triggers so they aren't run on this change? I think this is a behavior regression vs the previous test configuration. |
|
@pshapiro4broad - This has been the case for a while - We're in a tricky spot b/c we require tests to run in order for us to be able to merge the PR. So, if we skip the test run, then we're forced to override checks in order to merge the PR. |
Jira ticket: https://broadworkbench.atlassian.net/browse/DC-1227
Addresses
The TDR dev instance is more expensive than our staging and production instances. This is most likely due to our "personal" deployments. We also know that logging is one of our biggest costs of our infrastructure. We can see that our personal deployments are creating a lot of logs. So, by shutting them down, we expect the cost of the dev environment to decrease.
@fboulnois pointed out that we are seeing hundreds of thousands of logs per day on our personal environments:
Summary of changes
Testing Strategy
I followed these instructions to start up and shut down my own personal deployment.
Next Steps
Once this PR is approved, I will run
helmfile destroyon all of our remaining personal deployments. They can be recreated by runninghelmfile applyfrom the datarepo-helm-definitions repo as indicated in the instructions.