-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extending MongoDB auto-reconnect to all inserts (SCP-2629, SCP-5904) #379
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## development #379 +/- ##
===============================================
+ Coverage 75.79% 75.95% +0.15%
===============================================
Files 30 30
Lines 4470 4470
===============================================
+ Hits 3388 3395 +7
+ Misses 1082 1075 -7
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code looks good! Nice consolidation to the more robust approach.
# detect which argument is the list of inserted documents | ||
# when called from IngestPipeline, first arg is ingest instance, and 3rd is the list | ||
# when called from GeneExpression (static implementation), first arg is list | ||
if args[0].__class__.__name__ == 'IngestPipeline': | ||
docs_idx = 2 | ||
else: | ||
docs_idx = 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clever, nice solution.
Co-authored-by: Eric Weitz <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Verified ingest of a known good AnnData file behaves as expected (as does automated test).
Keeping my fingers crossed that this will do the trick for long subsampling jobs!
BACKGROUND & CHANGES
This update extends the current MongoDB "graceful auto-reconnect" functionality to all
insert_many
calls, not just those specific to gene expression data. This is in response to some instances ofConnection reset by peer
when ingesting other file types, such as metadata or subsampling of very large files. Additionally, now all database connections from ingest use theMongoConnection
module, standardizing howscp-ingest-pipeline
connects to database instances (as outlined in SCP-2629).MANUAL TESTING
This is (sadly) nearly impossible to test, as it requires a file that is not an expression matrix that will reliably trigger the
Connection reset by peer
error. Testing with the fragment files from SCP2805 will take multiple days to encounter the error (original job failed after ~5 days of runtime), if it happens at all.There is an automated test
test_insert_reconnect
intest_ingest.py
that mirrors the reconnection test in theGeneExpression
module. This test validates that when either theAutoReconnect
orBulkWriteError
errors occur, the reconnection code fires correctly. You can run this locally by initializing your ingest environment and then running this command from thetests
directory:You should see output similar to this:
Another manual test would be to use the ingest image
gcr.io/broad-singlecellportal-staging/scp-ingest-jb-mongo-retry-all:0f9452d
in your local SCP instance and validate that normal file uploads can write to the database as usual. Since all DB connections are now standardized and use the same reconnect code, we can at least prove normal operation is uninterrupted.