Handle invalid obsm keys with actionable error message (SCP-5496)

[jlchang]

Background

Previously when a user enters an incorrect obsm_key, AnnData ingest failed with a KeyError, resulting in “Unexpected exit status 1”. With the switch Batch API, the KeyError messaging is swallowed entirely, obscuring the cause of the failure for the study owner and makes troubleshooting take longer for an SCP admin.

Manual testing

1. Set up for ingest testing as per usual (from ingest directory, run source ../scripts/setup-mongo-dev.sh)
2. Run the following ingest pipeline job:
   python ingest_pipeline.py --study-id 5d276a50421aa9117c982845 --study-file-id 5dd5ae25421aa910a723a337 ingest_anndata --extract "['cluster']" --ingest-anndata --anndata-file ../tests/data/anndata/anndata_test.h5ad --obsm-keys "['foo']"
3. Confirm that user.log contains

Unable to extract cluster data from anndata file. Please check the provided obsm key, 'foo'.

[optional]

1. Boot as normal in your local instance
2. Set the ingest docker image for your local instance to: gcr.io/broad-singlecellportal-staging/scp-ingest-jlc_report_obsm_error:d20f5b8
3. Run an anndata ingest, supplying an invalid obsm key
4. Confirm that the ingest error email now clearly indicates the issue. Error message should be similar to:

"Unable to extract cluster data from anndata file. Please check the provided obsm key, 'foo'."

[codecov]

Codecov Report

Attention: Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.

Project coverage is 75.95%. Comparing base (9c81680) to head (d981059).
Report is 5 commits behind head on development.

Files with missing lines	Patch %	Lines
ingest/ingest_pipeline.py	76.92%	3 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           development     #390      +/-   ##
===============================================
- Coverage        75.97%   75.95%   -0.02%     
===============================================
  Files               30       30              
  Lines             4528     4538      +10     
===============================================
+ Hits              3440     3447       +7     
- Misses            1088     1091       +3     

Files with missing lines	Coverage Δ
ingest/cli_parser.py	82.96% <100.00%> (+0.12%)	⬆️
ingest/de.py	84.64% <ø> (ø)
ingest/ingest_pipeline.py	57.93% <76.92%> (+0.19%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eweitz]

Useful robustness improvement!