Spectrum sonification component #37
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR has a POC of a component for displaying a "sonified" version of a spectrum (technically it could be any 2D data, but spectra are the use case in mind here).
The basic building blocks here are that we use a Chart.js graph to display the data, and then listen for mousemove events to play the tone that corresponds to the x-value of the cursor (when the cursor is over the graph). Currently the audio implementation is straightforward and done using the Web Audio API.
The two important props that get passed into this component are
spectrum
contains the spectrum data as a list of (x, y) points. We can potentially refine the data structure for the spectrum later onsonifier
is "where the magic happens" - it's a function that takes in an (x, y) point (when called, this will be the point corresponding to the horizontal position of the mouse cursor), and returns a pitch and volume. The idea here is that a user can provide the way that they want to sonify the data here, so that the data -> sound mapping is configurable. Currently we just play a single output tone without any considerations for timbre, etc - that can potentially come in the futureOther things to note:
SpectrumSonifier
Storybook story on this branch (runyarn storybook
). The data there is one of the galaxy spectra that we used in the Hubble data story. Note that you need to press the "Start" button first, as most browsers require a user activation event in order to play sound via the Web Audio API.