Very large data sets and graphical performance

[stereokai]

1. Can you render very large datasets with Plot?
2. Can you render using WebGL?

Couldn't find these answers online

Thanks!

[Fil]

1. should be possible soon; @yurivish has been working on a Plot plugin that renders "density plots" (https://observablehq.com/collection/@twitter/density)
2. it's certainly possible to extend a mark with a custom renderer that returns a canvas handled by webgl, but it hasn't been done yet afaik. here's a similar approach with a canvas 2d renderer: https://observablehq.com/@observablehq/plot-canvas-dot; let me know if you want to collaborate on a webgl renderer.

[stereokai]

Thanks so much for the offer. I would love to collaborate but currently on a tight schedule for this project.

Those Density plots look downright impressive. He was able to attain this performance using just canvas? Big win. Either way, our use case is with line charts – specifically, displaying multiple line charts with different y scales but a synced x-axis (for simultaneous zooming and panning) and upwards of a million points per chart. Do you think something like the below mockup is possible today using Plot?

[enjalot]

In terms of "one million points per chart" you will likely want to do some kind of pre-processing as there aren't enough pixels to display a million points in a line chart (typically you have somewhere between 640-2000 pixels in the x-axis, so you could only show that many points).

What you probably want is some kind of strategy for sampling or aggregating before rendering the line. This something you can do in javascript, or with Plot (for example, Plot.bin supports time-based binning) and I would expect it to be performant with that much data.

[stereokai]

Of course, there aren't enough pixels to display that many points :)
However, we are optimistically hopeful that the downsampling will get taken care of by the charting library. So far, most of the libraries we tried did downsample internally pre-rendering. However, despite the downsampling, they all struggled with zooming and panning except for 1 commercial solution.

[enjalot]

Ah, I see what you mean. It may be helpful to not think of Plot as a charting library in the same vein as what you've likely evaluated. It's higher level than D3 but still requires you to wire up interactions and data transforms. For example, the brushing interaction you want to implement would still need to be built outside of Plot (unless you were to use / write a plugin) and the logic for updating the various line charts would also need to be written outside of Plot.

So your mock could be built with Plot today, but it would require a bit of programming to bring it together.

You may be interested in this article in particular (showing a brush that updates a line Plot) as well as the series its a part of:
https://observablehq.com/@observablehq/discovering-date-patterns?collection=@observablehq/analyzing-time-series-data

[stereokai]

I see. I'll check that series out.

Thanks for the heads up on that. In your experience, how involved do you reckon it would be to develop this UI with just the synced charts (in terms of zoom/pan) including the ability to update/replace the chart data, and to add & remove charts dynamically? Thanks.

[setvik]

they all struggled with zooming and panning except for 1 commercial solution.

@stereokai Out of curiosity, what was the commercial solution you found that didn't struggle?

[leeoniya]

canvas alone can get you quite far if the lib does minmax or lttb decimation. e.g. https://leeoniya.github.io/uPlot/bench/uPlot-10M.html