Skip to content

Spectrum sonification component #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

Carifio24
Copy link
Member

@Carifio24 Carifio24 commented Apr 23, 2025

This PR has a POC of a component for displaying a "sonified" version of a spectrum (technically it could be any 2D data, but spectra are the use case in mind here).

The basic building blocks here are that we use a Chart.js graph to display the data, and then listen for mousemove events to play the tone that corresponds to the x-value of the cursor (when the cursor is over the graph). Currently the audio implementation is straightforward and done using the Web Audio API.

The two important props that get passed into this component are

  • spectrum contains the spectrum data as a list of (x, y) points. We can potentially refine the data structure for the spectrum later on
  • sonifier is "where the magic happens" - it's a function that takes in an (x, y) point (when called, this will be the point corresponding to the horizontal position of the mouse cursor), and returns a pitch and volume. The idea here is that a user can provide the way that they want to sonify the data here, so that the data -> sound mapping is configurable. Currently we just play a single output tone without any considerations for timbre, etc - that can potentially come in the future

Other things to note:

  • I set this up using Chart.js (as opposed to e.g. Plotly) as it allowed easy access to mouse events. If we want to use another charting library I'm not opposed - I deliberately didn't go too far down the chart customization route
  • If you want to see it in action, go to the SpectrumSonifier Storybook story on this branch (run yarn storybook). The data there is one of the galaxy spectra that we used in the Hubble data story. Note that you need to press the "Start" button first, as most browsers require a user activation event in order to play sound via the Web Audio API.

@Carifio24 Carifio24 linked an issue Apr 23, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a component for spectral sonification
1 participant