🐛 Out of Memory crash during Release Build process #170
Description
Describe the bug
Running the Release Build process can cause the application to crash due to an out of memory error. The build also consumes the main Node thread making the application unresponsive to inbound HTTP requests including liveness probes.
The cause of these two issues needs to be determined through a review of the code, and then refactored so that the release build process does not cause crashes.
Potential Causes
- Song Analyses requests are being cached in memory to prevent duplicate calls, this could be loading memory with the entire song analyses data set
- considering moving storage out of memory to a key value store (redis/keyval)
- File/analyses data might be processed all at once instead of in batches, could reduce in memory workload by splitting this work into pieces
Expected behaviour
File indexer should be able to handle all data we expect to collect in the Toronto ARGO RDPC. Assuming 10k donors in the near term with ~10 analyses per donor, the indexer needs to handle 100k analyses (~300k files).