Loading metadata with p-limit
#103
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is build on top of:
music-metadata-browser
tomusic-metadata
#89I took a bunch of metadata files, spin op the audioMotion.js in server mode.
With the changes of #89 it metadata loaded slowly, massive parallel loading of the playlist, which took ~6 seconds to load all metadata.
First I disabled parallel fetching in the
addSongToPlayQueue
,retrieveMetadata
, function chain.The parallel loading is fact not done in
retrieveMetadata
, which is kind of mind fuck, as this method is responsible for the entire queue. Yet if a bunch of files are added to the queue, this function called for each item again and again. Hence #89 makes the parallel loading worse, and therefor performance goes down the drain.After disabling all parallel processing, putting
await
s all over the place. The increased the performance to ~1100 ms!Then I re-enabled parallel loading, now using
p-limit
, to avoid the number of parallel processing, and to slowly tear down the requirement to keep callingretrieveMetadata
way to ofton. Resulting to loading the metadata in 600ms.Technical debt
audioMotion.js/src/index.js
Line 1258 in 471e947
I maybe I should say, I kept as is on
dev
branch.There are several ways to resolve this, but it requires some refactoring.
One solution could be to push things which require metadata in a queue. Then we consume this queue by the the metadata
retrieveMetadata
, which maybe could be a worker / parallel processing thingy.Alternative solution
Update: I created #104 as probably a better alternative them, as that one queues items