Skip to content

Add STDOUT logging handler for Cloud Logging integration (SCP-5902) #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 24, 2025
Merged

Add STDOUT logging handler for Cloud Logging integration (SCP-5902) #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
129197b
adding STDOUT logging handler
bistline Jan 30, 2025
2278652
fixing typo
bistline Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ingest/de.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,7 +367,7 @@ def write_de_result(adata, group, annotation, rank_key, cluster_name, extra_para
clean_group = DifferentialExpression.sanitize_string(group)
out_file = f'{cluster_name}--{clean_annotation}--{clean_group}--{annot_scope}--{method}.tsv'
DifferentialExpression.de_logger.info(
f"Writing DE output for {clean_group} vs restq"
f"Writing DE output for {clean_group} vs rest"
)
elif de_type == "pairwise":
# rank_genes_groups accepts a list. For SCP pairwise, should be a list with one item
Expand Down
3 changes: 3 additions & 0 deletions ingest/monitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import logging
import os
import sys
from contextlib import nullcontext

# General docs for Sentry:
Expand Down Expand Up @@ -41,6 +42,8 @@ def setup_logger(logger_name, log_file, level=logging.DEBUG, format="default"):
logger.propagate = True
logger.setLevel(level)
logger.addHandler(handler)
# also log to STDOUT to pass to cloud logging
logger.addHandler(logging.StreamHandler(sys.stdout))
return logger


Expand Down
15 changes: 13 additions & 2 deletions scripts/vault-gsm-migration.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ function main {
BASEPATH="secret/kdux/scp/development/$(whoami)"
VAULT_SECRET=""
ROLLBACK="false"
while getopts "t:p:b:s:rh" OPTION; do
NAME_OVERRIDE=""
while getopts "t:p:b:s:n:rh" OPTION; do
case $OPTION in
t)
VAULT_TOKEN="$OPTARG"
Expand All @@ -55,6 +56,10 @@ function main {
VAULT_SECRET="$OPTARG"
echo "setting VAULT_SECRET to $VAULT_SECRET"
;;
n)
NAME_OVERRIDE="$OPTARG"
echo "using $NAME_OVERRIDE as GSM secret name"
;;
r)
ROLLBACK="true"
;;
Expand All @@ -78,7 +83,12 @@ function main {

# set up for GSM secret based on existing vault path
# changes / to -, e.g. mongo/hostname becomes mongo-hostname
GSM_SECRET_NAME=$(echo "$VAULT_SECRET" | sed 's/\//\-/g')
GSM_SECRET_NAME=$(echo "$VAULT_SECRET" | sed 's/[/_]/\-/g')

# use name override if specified
if [[ -n "$NAME_OVERRIDE" ]]; then
GSM_SECRET_NAME="$NAME_OVERRIDE"
fi

if [[ "$ROLLBACK" = "true" ]]; then
echo "Rolling back GSM migration"
Expand Down Expand Up @@ -109,6 +119,7 @@ USAGE:
-s VAULT_SECRET set the name of the requested secret in vault to copy (no default)

[OPTIONS]
-n GSM_NAME set the name of the new GSM secret, otherwise defaults to old name with special chars converted to -
-p PROJECT set the GCP project in which to create secrets (defaults to '$(gcloud info --format="value(config.project)")')
-b BASEPATH set the base vault path for retrieving secrets (defaults to 'secret/kdux/scp/development/$(whoami)')
-r roll back migration and delete scp-ingest-pipeline secret in GSM
Expand Down