Changed the MaskEditor to an Image Canvas

[brucew4yn3rp]

#2888

Revamped the MaskEditor UI and changed it to an Image Canvas. The context menu name has also been renamed accordingly.

The interface now supports drawing on the image itself (on the image layer) in addition to covering the area with a mask. Users can select a color or use an eyedropper tool for color matching. As indicated in the Feature Request, this is a powerful tool to guide the image generation process post-masking.

The tools on the left sidebar have been updated accordingly. The top one, is the existing mask brush, and has a matching mask logo. The second one is the image painting brush.

┆Issue is synchronized with this Notion page by Unito

[trsommer]

this is an interesting idea. But I just tried it and it does not seem to work? I drew on the image and saved my progress, but my drawing is not saved to the image. Also, I can't erase my drawing. You should also separate the drawing into its own layer so the user can toggle their drawing on/off.

[socket-security]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher

View full report↗︎

[brucew4yn3rp]

@trsommer It took me a few more hours to understand the intricacy of your MaskEditor code.

The updated version should now paint on a separate layer (with a checkbox for visible / invisible). You are also now able to erase your painted features. When using the paintbrush, you can right click and draw to erase just painted features. When using the mask, you can right click and draw to erase just mask area. When using the eraser tool, it erases both.

Also, it now saves properly.

[trsommer]

First of all, I appreciate you understanding my confusing mask editor code. It needs further refactoring. As you said, the saving works, and you can erase using the eraser tool.

There are a few more bugs that need fixing:

• The line eraser is triggered when erasing using the right mouse button while using the draw tool.
• The brush hardness for the mask tool is broken; the brush tool does not have it.
• Undo/redo does not work for the draw tool.
• I need a way to only erase the drawing or the mask part. The easiest solution would be to make the layer clickable to select which layer to work on.
• (There are UI changes I would like to make, but that is not important for now.)
• you need to adhere to the prettier code style
• Last but not least, I feel like the name change in the menu is confusing. If I am accustomed to the mask editor, I will not find it now. The name 'Image Canvas' does not convey mask editing to me. I think we should stay with 'Mask Editor' because mask editing is the main feature for now. In the future, I think the mask editor and a potential image editor should be split.

[brucew4yn3rp]

@trsommer Thank you for the comments. I've put comments below, but most importantly, in the prior update, I was flattening the rgbCanvas and the base imageCanvas into one image (and using that as the ref for the mask) in order to re-use the mask loading structure. I've since removed that functionality because it does not allow the user to re-edit their painting after saving (since the painting is flattened with the image). I would like to save the painting (rgbCanvas) separately, like the mask is saved separately, to retain the editability.

The line eraser is triggered when erasing using the right mouse button while using the draw tool.

• This is fixed

The brush hardness for the mask tool is broken; the brush tool does not have it.

• This is fixed

Undo/redo does not work for the draw tool.

• This should be fixed

I need a way to only erase the drawing or the mask part. The easiest solution would be to make the layer clickable to select which layer to work on.

• Layer buttons have been implemented. Also, right click will only apply to erasing whichever tool is activated. Additionally, the eraser tool will only erase on the active layer. Selecting the mask tool automatically activates the mask layer, and selecting the Paint tool automatically activates the Paint layer.

you need to adhere to the prettier code style

• I wasn't aware of the prettier code style, apologies.

Last but not least, I feel like the name change in the menu is confusing. If I am accustomed to the mask editor, I will not find it now. The name 'Image Canvas' does not convey mask editing to me. I think we should stay with 'Mask Editor' because mask editing is the main feature for now. In the future, I think the mask editor and a potential image editor should be split.

• I discussed this with @LukeG89 when I first created the Feature Request. Maybe a better name is "Mask Editor | Image Canvas" but I'm not too invested in the "Image Canvas". However, I believe that it is important to keep the mask editor and "image editing" together because the primary use-case is that users can mask off a painted area with low opacity to guide downstream sampling.

[brucew4yn3rp]

@trsommer I've pushed an update - the code now saves the rgbCanvas pixels to the input folder.

So, now the code saves the mask image, a combined image with the mask and painted pixels, and an image with just the paint pixels.

We still need to figure out how to load the painted pixels onto rgbCanvas when the MaskEditor is re-opened after painting and saving. After the initial painting and save, the node will process the combined image, meaning both the mask and the painting will get passed on. But, when the MaskEditor is re-opened (for further adjustment of mask/paint), the paint does not show up.

[brucew4yn3rp]

@trsommer Have you had a chance to take a look? I think this is a very useful tool to have, but I'm currently stuck at being able to load the painted pixels because I'm not sure how to replicate the clipspace behavior that the mask image uses.

It would be immensely helpful if you could help bring this over the finish line given your familiarity.

[trsommer]

@brucew4yn3rp Unfortunately, I haven't had a chance yet. I'm quite busy for the next two weeks, but I will look at it after that

[brucew4yn3rp]

Thank you @trsommer. I've just pushed an updated file that adds an index to the Clipspace type, which allows for the persistence of the painted pixels. Now after the maskeditor is closed and re-opened, painted pixels are still editable.

The only thing left to figure out now is how to pass the painted pixels + base image as the image in the node. I had some luck doing this with a combinedCanvas that draws the image and rgbCanvas but then we lose the mask.

Whenever you have a chance, if you can take a look at that, I think that will be the last step to finalize this.

[brucew4yn3rp]

Hi @trsommer - hope the past two weeks went well. Quickly following up on the final stage of the MaskEditor upgrade whenever you have a chance.

Thank you.

[trsommer]

@brucew4yn3rp I looked into the issue. I honestly have no idea how to solve it. I achieved the same result as you, retaining the paint and mask layers, but I did not figure out how to output the painted image while retaining the original input image with mask and paint layers. I will try again tomorrow.
For context, I didn't create the send/receive logic for the mask editor; it was largely already in place when I developed MaskEditor 2.0