Best data source for periodically growing lines in a PathLayer?

[hrbigelow]

I have an application which periodically produces new batches of points for a number of existing lines. These lines can get quite long though (upwards of 1M points) although the total number of lines in a plot is modest (less than 10). A typical run may produce 100 new points every second or so.

At the moment, I was thinking to use PathLayer with:

function onDeviceInitialized(device) {
  const buffer = device.createBuffer(new ArrayBuffer(...))
  const layers = [
    new PathLayer({
      data: {
        length: 10,
        startIndices: [0, 1000, 2000, 3000, ...],
        attributes: {
          getPath: { buffer, type: 'float32', size: 2, offset: 0, stride: 2 }
        },
      }  
    });
  ]
  deck.setProps({layers})
}

but, then when new data arrives, unfortunately, since all of the data for any given line needs to be contiguous, I need to rewrite the entire buffer data.

For example, if my buffer contains data for lines A, B and C, and it initially looks like:

AAAABBBBCCCCCCC

then I receive new data AA, B, CCC, then I need to rewrite the whole thing as

AAAAAABBBBBCCCCCCCCCC

Is there any better way to do this in this situation?

Here is an example of data accumulated over about an hour, with two lines and 1M points per line (log log plot). This example is using Bokeh, but I wondered if deck.gl could be more flexible and performant.

[chrisgervang]

Have you considered using one PathLayer per line, and then maintaining one buffer per line?

Usually it's recommended to use one layer, as you've done, but anything under 256 should be fine (I think 256 is a soft cap and primarily matters when the app needs picking interactivity)

[hrbigelow]

Ahh, no I hadn't thought of that. In the case of 10 lines x 10 million points / line, that does seem like a good strategy. Thank you!

[hrbigelow]

One question: how hard would it be to extend the data attributes mechanism to include endIndices (or, numElements) alongside the startIndices mechanism for packing multiple lines' data into a single buffer? - that way, one could do an on-demand periodic doubling reallocation of an underlying buffer.

[chrisgervang]

You might also consider Perspective for this use case. It focuses more on realtime traditional charting for large datasets.

[hrbigelow]

ahh thanks - it doesn't look like it can do 3D though? At least they don't seem to show that in the examples. (I'm hunting for something very general as I'll be using it for both 2D line, scatter, 3D scatter, and possibly other kinds of visualizations)

[chrisgervang]

Yeah, I'm not sure. Could be worth asking them

[hrbigelow]

I looked over the Perspective repo but it doesn't look appropriate for my use case. There is no GL rendering and no examples of 3D point clouds. I'll noodle around a bit. Your feedback was helpful.