Feature Proposal: Voxel Annotation Layer

[briossant]

Hello everyone,

I would like to propose a new feature: an interactive voxel annotation layer for manual segmentation directly within Neuroglancer. I am working at Ariadne.ai, and this capability would be a great addition to our platform and, I believe, to the broader Neuroglancer community. I also saw that this feature was mentioned in the following talk: https://www.youtube.com/watch?v=_XgfGcu81AA

I am aware that the contribution guidelines recommend discussing large features before implementation. To better understand the technical challenges, facilitate a more concrete discussion and familiarize myself with neuroglancer, I have built a prototype (see https://github.com/briossant/neuroglancer/tree/feature/voxel-annotation). I'm opening this issue now to share the proposed architecture and ask for the community feedback.

DEMO video: https://youtu.be/7d_PiqV-h2w

Proposal

The core idea is a new layer type, vox, that allows users to paint voxel values directly onto a volume. My prototype implements the full end-to-end workflow, from UI interaction to data persistence and multi-resolution updates.

Here is a breakdown of the proposed architecture:

1. Frontend Components (layer/vox/, ui/voxel_annotations.ts)

• VoxUserLayer: A new UserLayer subclass that orchestrates the feature.
• Drawing Tools (VoxelBrushLegacyTool, etc.): These tools capture mouse events. On mousedown/drag, they calculate the target voxel coordinates based on the current view and MouseSelectionState.
• Frontend VoxelEditController: The tools call methods on this controller (e.g., paintBrushWithShape). This frontend controller is responsible for:
• Optimistic Local Preview: It immediately applies the edit to the in-memory VolumeChunk data (source.applyLocalEdits). This provides instant visual feedback to the user without waiting for the backend. The modified chunks are marked for GPU re-upload.
• Backend Communication: It commits the edits to its backend counterpart via an RPC (commitEdits).

2. Backend Controller (voxel_annotation/edit_backend.ts)

• VoxelEditController (Backend): A SharedObject on the web worker that serves as the authoritative state manager.
• Edit Batching: It receives edits, debounces them, and aggregates them into a single transaction to minimize write operations.
• Data Persistence: It calls a new applyEdits method on the appropriate VolumeChunkSource backend. This method handles the actual writing to the data store (e.g., Zarr, ...). It returns the original voxel values that were overwritten.
• Undo/Redo History: Upon a successful write, it stores the returned VoxelChange (containing indices, old values, and new values) in an undoStack. It also manages the redoStack.
• Downsampling Orchestration: After a successful commit, it adds the key of the modified chunk to a queue for downsampling. After each downsampling level is done, an RPC (callChunkReload) is sent to the frontend to invalidate the now-stale chunks, forcing a re-fetch from the data source.

Code Flow for a Single Brush Stroke

This diagram illustrates the sequence of events from a user drawing on the canvas to the data being persisted and downsampled.

sequenceDiagram
    participant User
    participant Tool as VoxelBrushLegacyTool
    participant ControllerFE as VoxelEditController (Frontend)
    participant SourceFE as VolumeChunkSource (Frontend)
    participant ControllerBE as VoxelEditController (Backend)
    participant SourceBE as VolumeChunkSource (Backend)

    User->>Tool: Mouse Down/Drag
    Tool->>ControllerFE: paintBrushWithShape(mouse, ...)
    note right of ControllerFE: Calculates affected chunks/voxels

    ControllerFE->>SourceFE: applyLocalEdits(chunkKeys, ...)
    SourceFE->>SourceFE: Modifies in-memory chunk data
    note right of SourceFE: Viewer redraws with updated chunk data (preview)
    SourceFE->>ChunkManager: Invalidates GPU texture for chunk

    ControllerFE->>ControllerBE: commitEdits(edits, ...) [RPC]

    activate ControllerBE
    ControllerBE->>ControllerBE: Debounces and batches edits
    ControllerBE->>SourceBE: applyEdits(chunkKeys, ...)
    activate SourceBE
    SourceBE-->>ControllerBE: Returns VoxelChange
    deactivate SourceBE

    ControllerBE->>ControllerBE: Pushes change to Undo Stack
    ControllerBE->>ControllerBE: Enqueues chunk for downsampling
    deactivate ControllerBE

    loop Downsampling Cascade
        ControllerBE->>ControllerBE: downsampleStep(chunkKeys)
        ControllerBE->>SourceBE: applyEdits(chunkKeys, ...)
        note right of SourceFE: Reload chunks affected by the downsampling
        ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
        activate ControllerFE
        ControllerFE->>SourceFE: invalidateChunks(chunkKeys)
    end
    deactivate ControllerFE

Loading

Architectural Discussion Points

1. Low res drawing / big brush

The prototype currently locks drawing to the highest resolution level. An attempt was made to allow drawing at lower resolutions, based on the brush size. However, the implementation of an upscaling cascade is not trivial; two approaches were considered:

• first to mimic the downscaling process, but with the exponential nature of the upscaling, we quickly hit the limitation of the number of upscale steps we can perform.
• second, to delay the upsampling to the moment we need the chunk, but this method is not compatible with generic data sources and requires a complex logic to handle conficts.

A solution would be to allow the low-resolution drawing while accepting that no upscaling will be performed. This would allow the feature to be used to outline datasets for example, but ui work is needed to avoid user confusion.

2. Live Preview on Compressed chunks

The applyLocalEdits() method inside the VolumeChunkSource is writing the edits directly into the chunk in memory, which is not possible for compressed chunks, as far as I know. To temporarily solve this, the compressed chunk are decompressed, edited and then re-compressed, but this obviously leads to poor performance. The only solution I have currently in mind would be to move the preview in a separate overlay that the render layer would render on top of the original chunks.

3. Annotating over Read-Only Sources & multi-datasources behavior

The current implementation was thought with a single source in mind. But use cases like annotating over a read-only source require considering a multi-datasources behavior. The trivial way would be to read and try to write to every source and then mix the chunks in the shader. But I think it would be better to have a priority system where we read through the sources until we find one that has the wanted chunk, and we would write to the most prioritized source or let the user choose the writable sources.

4. Dataset Creation

I think having the ability to create new datasets within neuroglancer would be a great feature, this would prevent the need of external tools or a dedicated backend, especially for the case of writing over read-only sources. This could be an option proposed to the user when the datasource url link to an empty directory for example. We would then let the user choose the dataset format, bounds, resolution, etc.

Looking forward to any feedback, Thank you.

[jbms]

This is awesome work! Let's definitely work to get this feature merged!

I so far have only taken a quick look at your branch, but I have a few comments:

• Rather than have a separate layer type, I think the drawing tools could just be supported on image and segmentation layers that have a writable volumetric source.
• Rather than use the "legacy tool" mechanism, it should use the new tool mechanism. You could make the tools "toggle" tools, so that they stay activated, in order to make them more convenient. There have been separate discussions about making the new tool mechanism more ergonomic: fix(datasource/graphene) swap selected layer when activating merge tool #821 (comment)

To support low res drawing efficiently, ultimately we need a mechanism to avoid having to upsample most or all of the time. To accomplish that will, I think, inherently require a new format (some sort of editable multiscale format based on zarr, perhaps). Broadly I think it would need to allow you to have multiple overlaid chunks at a given resolution and position, where each overlay chunk has:

• a timestamp, probably encoded in the filename/key itself, to order them correctly
• a binary mask indicating which voxels are updated by the overlay
• new values for positions within the binary mask
  Then to read at a given resolution R, you have to consider all overlaid chunks at resolution R and at coarser resolutions, but you don't have to consider chunks at finer resolutions.

Supporting existing formats would be great also --- perhaps it would just operate in a degraded/less efficient mode when using existing formats.

Undo/redo is a pretty critical feature --- perhaps being purely in the local browser state is sufficient. Ideally it could be integrated into the format itself so that it can support multiple users, and undo changes not made in the current browsing session. Possibly that could already be supported by using OCDBT as the kvstore.

[fcollman]

i just wanted to echo jeremy's sentiments and enthusiasm. This is a very cool direction and I hope we find a way toward integrating this.

One thing I was curious about is whether you have experimented with 3d drawing or just 2d drawing. In other tools, drawing in 3d efficiently has a number of optimizations that aren't relevant for 2d drawing, such as interpolation between labels drawn across different sections, 3d filling, etc etc. Generalizing this to Nd might also be challenging (i.e. interpolation of 3d drawing across timepoints).

[briossant]

Thanks Jeremy and Forrest for the enthusiasm and feedback! That’s really encouraging!

@fcollman --- I haven’t experimented with 3D drawing yet. This voxel annotation feature is quite large in scope, so I had to choose what to focus on first, but 3D drawing could definitely be a nice addition down the line. The other software we use at Ariadne.ai (Knossos) supports it, so it’s an interesting reference point.

Regarding the first two suggestions from @jbms --- they make total sense. Just to clarify a bit:

• When merging voxel annotation with the segmentation and image layers, we’d want slightly different behavior between the two. For the segmentation layer, the idea would be to use the existing segment (which I called labels in my prototype) management, adding a way to create new segments. For the image layer, the user could directly choose the value/color to draw with. We could probably reuse a similar mixin structure as the polygonal annotations to make this work.
• For the switch to the new tool system, we definitely want to keep the ctrl+click behavior for the brush and flood fill tools, so the "toggle tool" you mentioned seems like the best way to go (if I understand what you mean the right way).

More broadly, it might be helpful to define what we’d like to include in the first version of this feature. I have about 3.5 months left at Ariadne.ai before my internship ends, after which I’ll return to school for my final year --- and time will be more limited then.

I do think that supporting other formats is nice. However, in that case, I do not see any way of allowing truly efficient low-res drawing; it can work for 2-3 steps max. But at the same time, I don't really see the cases where someone would need to draw huge areas and tiny details at the same time, I might be unaware.
But either way having a true multi-res drawing would be really nice, I just do not know if it is best to start with that now or wait until we have a solid voxel annotation feature.
In an earlier version of the prototype, I was saving the drawing in the local browser storage, which allowed for custom formats, so I tried to implement an upsampling pipeline which was heading to something kinda similar to what you described, but I scrapped it before it worked to refocus on what was the MVP.

Finally, for the undo/redo, if we also want the persistant for every format, we might consider using local storage (indexedDB or OPFS).

Also, how do you usually handle communication and workflow for Neuroglancer development?