Add STDOUT logging handler for Cloud Logging integration (SCP-5902)

[bistline]

BACKGROUND & CHANGES

This update adds a logging handler for STDOUT to supplement the existing logging for ingest processes. By adding this, we will leverage the existing integration of cloud logging into Batch API jobs. This will add much greater visibility for jobs as we will not need to rely on the log file being delocalized back to the bucket. This benefits running jobs the most as we will get real-time log messages, allowing us to debug the state of a job without waiting for completion.

Note: This also contains an unrelated update to the vault-gsm-migration.sh script that was needed to help import secrets from Vault into GSM better.

MANUAL TESTING

1. Initialize your environment as normal and run the following command from the ingest folder:

python ingest_pipeline.py --study-id 5d276a50421aa9117c982845 --study-file-id 5dd5ae25421aa910a723a337 ingest_cluster --cluster-file ../tests/data/test_1k_cluster_data.csv --ingest-cluster --name cluster1 --domain-ranges "{'x':[-1, 1], 'y':[-1, 1], 'z':[-1, 1]}"

2. In addition to normal output, you should see messages that normally would only print to user_log.txt:

Transforming header CLUSTER
Transforming header SUBCLUSTER
Transforming header 1
Creating model for 5d276a50421aa9117c982845
Creating data array for header NAME
Attempting to load cluster header : NAME
Creating data array for header x
Attempting to load cluster header : x
Creating data array for header y
Attempting to load cluster header : y
Creating data array for header z
Attempting to load cluster header : z
Creating data array for header CLUSTER
Attempting to load cluster header : CLUSTER
Creating data array for header SUBCLUSTER
Attempting to load cluster header : SUBCLUSTER
Creating data array for header 1
Attempting to load cluster header : 1

[eweitz]

Code looks good!

Are there use cases that old file-based logging satisfies that this new stdout + cloud logging doesn't also satisfy? At a glance, this new approach seems like it might make that prior one redundant.

[bistline]

Code looks good!

Are there use cases that old file-based logging satisfies that this new stdout + cloud logging doesn't also satisfy? At a glance, this new approach seems like it might make that prior one redundant.

That's a good point. Let me do a little testing but you might be right.

[bistline]

Code looks good!
Are there use cases that old file-based logging satisfies that this new stdout + cloud logging doesn't also satisfy? At a glance, this new approach seems like it might make that prior one redundant.

That's a good point. Let me do a little testing but you might be right.

Just to close the loop on this - the one place where we still need the physical log file is when Rails reads it to send an error email to users. Because we can't get the Cloud Logging client for Ruby to work properly, this is the only way to get that information.