Visualizer Refactor

[RoboDoig]

This PR refactors the dynamic foraging visualizers to use a full Bonsai ImGui approach according the the requirements outlined in the visualizer refactor milestone

Addresses issues #99, #100, #101, #102, #103, #104, #105.

Individual visualizers take as input an observable source that describes their parent layout group (e.g. tab area, table window) such that visualizers can be reorganised within a parent layout structure. Generally, data is mapped to visualizers by externalized subject names corresponding to the appropriate data source or event.

[micahwoodard]

Pushed some changes/fixes to update reward delivery changes in flush/open functions and allow for SessionInfoVisualizer to ingest a serializable object and display values. Also added stage name to metadata visualizer. A couple of issues that still need to be adressed

1. The bias graph time axis will jitter with just one bias point
2. when pressing stop, bonsai crashes

[micahwoodard]

Pushed more updates adding stage name to metadata bar and using trial index for bias

[micahwoodard]

Pushed fix for crashing behavior when ending session. Problem was arising when camera preview visualizer continued capturing images when camera stream was closed

[micahwoodard]

Experiencing bug where it seems like the response period is truncated somehow. When testing with new visualizers, mice received no rewards even though they were consistently licking. Upon further investigations, it seemed like mice were not licking within the response period but immediately after. Comparing the old vs new visualizers, it seems like the quiescent period is missing from the trial graph, but I suspect this is actually effecting the response deadline instead. When running mice with the old visualizers with identical parameters, we did not see mice missing response deadlines and consistently making choices at the very beginning of the response period.

These images were produced with identical task logics

[bruno-f-cruz]

@RoboDoig This looks great Andrew! thanks. I left a few notes after a quick review just to make sure we dont run into issues later!

[bruno-f-cruz]

@RoboDoig Some context is missing here: there was a reason the previous visualizer was done at the OpenGL layer. If I understand correctly, this observable is allocating 3 frames on each Select call. These cameras can be quite high frame rate (500fps), not sure this will scale. Have you benchmarked it?

[RoboDoig]

@bruno-f-cruz Thank you, good point! I will have a look, is resolution also variable on the cameras used?

[bruno-f-cruz]

In me experience yes. Not sure for this project in particular. But also not sure why avoiding the shader-based solution in the first place. I does not allocate, and handles textures in normalize coordinates so you don't have to worry about resolution. Also, it dynamically matches the frame rate of the visualizer since it runs on the draw call, not on each input.

[RoboDoig]

@bruno-f-cruz Can we also assume a grayscale image from the camera output? Or are there a mix of monochrome and color cameras?

[bruno-f-cruz]

I always default to using the first channel (could be exposed via a uniform if needed I guess). So for the most part sticking with mono is the intended way. I think this composes well since with color you risk hitting the blue and red tones and slicing a channel is 0-copy, if you need to select a specific color to threshold.

[RoboDoig]

I just did quick benchmark now and you're completely right, this method causes significant slowdown for a number of reasons. Image allocations is one, but I think also just the CV commands required to replicate the shader can't keep up with anything much over 60fps. There are ways to optimise this but I think it probably makes more sense to revert back to the shader approach and refactor to fit that in with the rest of the ImGui approach.

[bruno-f-cruz]

these files should be ignored and not committed

[bruno-f-cruz]

Delete the churn diff

[micahwoodard]

Experiencing bug where it seems like the response period is truncated somehow. When testing with new visualizers, mice received no rewards even though they were consistently licking. Upon further investigations, it seemed like mice were not licking within the response period but immediately after. Comparing the old vs new visualizers, it seems like the quiescent period is missing from the trial graph, but I suspect this is actually effecting the response deadline instead. When running mice with the old visualizers with identical parameters, we did not see mice missing response deadlines and consistently making choices at the very beginning of the response period.

These images were produced with identical task logics

@RoboDoig circling back to this issue. After further debugging this morning, I believe the camera visualizers are the root cause. The rendering overhead appears to be introducing enough latency that the task engine's lick detection is significantly delayed, causing responses to be recorded outside the response window despite occurring well within it.

I confirmed this by manually simulating licks while observing both camera frame lag and recorded licks in the ethogram visualizer: both were noticeably delayed. When I disabled the camera visualizers entirely, the bug disappeared. This is consistent with the performance concerns Bruno raised above.

[RoboDoig]

@micahwoodard - I've reimplemented the visualizer with a shader pass rather than OpenCV transformations in PR #162. I was not able to test this on a camera with frame rate >200Hz but it seemed to eliminate latency problems on the hardware I have. Would you be able to test on your rigs?

[micahwoodard]

Yes will try and test this afternoon