Skip to content

canvas2D float: Separate ImageData and update #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

canvas2D float: Separate ImageData and update #126

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
265 changes: 68 additions & 197 deletions canvas_float.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,15 @@

## Introduction

This proposal introduces the ability to use floating-point pixel formats in `CanvasRenderingContext2D`, `OffscreenCanvasRenderingContext2D`, and `ImageData`.
This proposal introduces the ability to use floating-point pixel formats in `CanvasRenderingContext2D` and `OffscreenCanvasRenderingContext2D`.

## Background

## Current capabilities

Both `CanvasRenderingContext2D` and `OffscreenCanvasRenderingContext2D` contain an output bitmap that they render to.
The pixel format of this output bitmap is currently unspecified.
Many implementations use a 8 bits per channel RGB or RGBA pixel format for this bitmap (this is likely the case for all implementations, but the author has not examined all implementations).

An `ImageData` has a `data` member, which is a `Uint8ClampedArray`, making its format 8 bits per channel RGBA.
The pixel format of this output bitmap is currently unspecified, but all implementations currently use a 8 bits per channel RGB or RGBA pixel format for this bitmap.

## Use Cases and Motivation

Expand All @@ -26,7 +24,7 @@ Modern high end displays are capable of displaying more than 8 bits per channel.

### Changes to `CanvasRenderingContext2DSettings`

Create the new enum `CanvasColorType` to specify the type for the color channels of the output bitmap of a `CanvasRenderingContext2D` and `OffscreenCanvasRenderingContext2D`.
Create the new enum type `CanvasColorType`.

```idl
enum CanvasColorType {
Expand All @@ -35,94 +33,53 @@ Create the new enum `CanvasColorType` to specify the type for the color channels
};
```

Add to `CanvasRenderingContext2DSettings` a `CanvasColorType` member, to specify this color type.
Add to `CanvasRenderingContext2DSettings` a `colorType` member of type `CanvasColorType`, with a default of `"unorm8"`.

```idl
partial dictionary CanvasRenderingContext2DSettings {
CanvasColorType colorType = "unorm8";
};
```

The value specified in `colorType` will determine the type for the color channels of the output bitmap.
### Changes to `HTMLCanvasElement` and `OffscreenCanvas` 2D context creation

When a canvas has a color type of `"float16"` color values may be outside of the range `[0, 1]`.
Such values may be used to specify colors outside of the gamut determined by the canvas' color space's primaries.
In
[`HTMLCanvasElement`'s 2D context creation algorithm](https://html.spec.whatwg.org/multipage/canvas.html#2d-context-creation-algorithm)
and
[`OffscreenCanvas`' 2D context creation algorithm](https://html.spec.whatwg.org/multipage/canvas.html#2d-context-creation-algorithm),
when the `CanvasRenderingContext2DSettings` is created (from the optional `options` parameter),
the value specified in `colorType` will determine the type for the color channels of the rendering context's [output bitmap](https://html.spec.whatwg.org/multipage/canvas.html#output-bitmap) as follows:
* If the color type of the output bitmap is `"unorm8"`, then the color channels of the be 8 bit unsigned.
* If the color type of the output bitmap is `"float16"`, then the output bitmap's precision must be IEEE 754 16-bit floating-point.

When rendering `"float16"` color values to an output device, color values will be converted to the output device's color space using relative colorimetric intent.
Colors values that specify a brightness outside of the standard dynamic range will have their brightness limited to the standard dynamic range of the output device, unless the canvas is explicitly indicated to be high dynamic range (and this proposal intentionally does not include this mechanism).
### Floating point behavior

### Changes to `ImageData`
Rendering operations to an output bitmap of color type `"float16"` may be subject to the same differences from IEEE 754 behavior that are outlined in [WebGPU's WGSL](https://www.w3.org/TR/WGSL/#differences-from-ieee754).

Create a new enum `ImageDataColorType` to specify the type for the color channels of an `ImageData`.
### Display of values outside of the `[0, 1]` interval

```idl
enum ImageDataColorType {
"unorm8",
"float32",
};
```

Add to `ImageDataSettings` a `ImageDataColorType` member, to specify this color type.

```idl
partial dictionary ImageDataSettings {
ImageDataColorType colorType;
};
```

Change `ImageData` to allow the `data` member to be either `Uint8ClampedArray` or `Float32Array` via a union type `ImageDataArray`.

```idl
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

partial interface ImageData {
readonly attribute ImageDataColorType colorType;
readonly attribute ImageDataArray data;
}
```

In the constructor for `ImageData`, if an `ImageDataSettings` is specified, and that `ImageDataSettings` has a `colorType`, then the created `ImageData` will have the specified value for its `colorType`.
If no `ImageDataSettings` is specified, or no `colorType` is specified, then the created `ImageData` will have `colorType` `"unorm8"`.

The type of an `ImageData`'s `data` member is determined by its `colorType` member as follows:

* If `colorType` is `"unorm8"` then `data` is of type `Uint8ClampedArray`.
* If `colorType` is `"float32"` then `data` is of type `Float32Array`.

### Changes to `CanvasImageData` interface

In the interface `CanvasImageData`, the functions `createImageData` and `getImageData` will create an `ImageData`, and also optionally take an `ImageDataSettings`.

If the `ImageDataSettings` specifies `colorType`, then the resulting `ImageData` will have that specified `colorType`.

If the `ImageDataSettings` does not specify `colorType`, then the resulting `ImageData`'s `colorType` will be `"unorm8"`.

### Values outside of the `[0, 1]` interval
When rendering a canvas to an output device, color values are [converted to the output device's color space using relative colorimetric intent](https://html.spec.whatwg.org/multipage/canvas.html#colour-spaces-and-colour-correction).

All color spaces supported by `PredefinedColorSpace` are defined for all real numbers.
Add text clarifying that colors values outside of the standard dynamic range of the output device will be projected to the standard dynamic range of the output device, as is [already specified in WebGPU](https://www.w3.org/TR/webgpu/#gpucanvastonemappingmode).

Both `"srgb"` and `"display-p3"` are extended using point symmetry around `0`.
For the precise definition, see [the CSS definition of `"srgb"`](https://www.w3.org/TR/css-color-4/#predefined-sRGB).
### Preservation of precision during export

### Type precision
#### To `ImageBitmap`

The exact storage format of the output bitmap of a `CanvasRenderingContext2D` and `OffscreenCanvasRenderingContext2D` is not directly observable.
There exists no API through which the raw storage may be examined.
When converting an output bitmap of color type `"float16"` to an `ImageBitmap` via [`createImageBitmap`](https://html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#dom-createimagebitmap), the resulting `ImageBitmap` must not lose any precision.

If the color type of the output bitmap is `"unorm8"`, then the precision must be at least 8 bits per color channel.
Stored values less than 0 or greater than 255 must be clamped to 0 and 255 respectively.
#### To encoded image

If the color type of the output bitmap is `"float16"`, then the precision must be at least IEEE 754-2008 16-bit floating-point.
When [serializing an output bitmap to a file](https://html.spec.whatwg.org/multipage/canvas.html#a-serialisation-of-the-bitmap-as-a-file) (e.g, using `toBlob()` or `toDataURL()`, the output file format will impose restrictions on what can be encoded. To that section, add the following clarifications:

### Type conversions
* For image types that do not support storing floating point values, pixel values will be clamped to the range `0.0` through `1.0`.
* For the `image/png` image type, if the output bitmap has color type `"float16"`, then the encoded PNG image must be 16-bit.

When reading an output bitmap of color type `"unorm8"` to a `Float32Array`, the resulting values will be normalized from the range `0` through `255` to the range `0.0` through `1.0`.
#### To `ImageData`

When converting an output bitmap of color type `"float16"` to a `Uint8ClampedArray`, values less than `0.0` will be clamped to `0`, values greater than `1.0` will be clamped to `255`, and values in the range `0.0` through `1.0` will be scaled by `255` and rounded to the nearest integer value.
When converting an output bitmap of color type `"float16"` to a `Uint8ClampedArray` via the `getImageData` method, pixel values will be clamped to the [0,1] interval, scaled by `255`, and rounded to the nearest integer value.

When exporting an output bitmap of color type `"float16"` to an data URL or a `Blob` in a format that does not support floating point storage, values outside of the range `0.0` through `1.0` will be clamped.

## Related specifications
## Related specifications and issues

### WebGPU

Expand All @@ -131,157 +88,71 @@ They may be specified in `GPUCanvasConfiguration` by indicating a `format` of `r

### WebGL

In WebGL, floating-point canvas color types are proposed via [drawingBufferStorage](https://github.com/KhronosGroup/WebGL/pull/3222).
In WebGL, floating-point canvas color types are already available.
They may be specified via `drawingBufferStorage` by indicating a `sizedFormat` of `RGBA16F`.

### Canvas High Dynamic Range

This functionality is a prerequisite for the [Canvas High Dynamic Range proposal](https://github.com/w3c/ColorWeb-CG/blob/master/hdr_html_canvas_element.md).

This functionality was separated off from the Canvas High Dynamic Range proposal.

## Examples

### Writing the same color in different ways

The following example draws the color `color(display-p3 1 0 0)` in multiple ways.

```javascript
let canvas = document.getElementById('MyCanvas');

let context = canvas.getContext('2d', {colorSpace:"display-p3"});
console.log(context.getContextAttributes().colorSpace) // Prints "display-p3"
console.log(context.getContextAttributes().colorType) // Prints "unorm8"

// Draw using CSS colors.
context.fillStyle = "color(display-p3 1 0 0)";
context.fillRect(0, 0, 1, 1);

// Draw using a floating-point sRGB ImageData, using values outside of the
// [0, 1] interval.
let imageDataSRGB = new ImageData(1, 1, {colorType:"float32"});
console.log(imageDataSRGB.colorSpace); // prints "srgb"
console.log(imageDataSRGB.colorType); // prints "float32"
imageData.data[0] = 1.0930663624351615;
imageData.data[1] = -0.22674197356975417;
imageData.data[2] = -0.15013458093711934;
imageData.data[3] = 1.0;
context.putImageData(imageDataSRGB, 1, 0);

// Draw using a floating-point Display P3 ImageData.
let imageDataDisplayP3 = new ImageData(1, 1, {colorSpace:"display-p3", colorType:"float32"});
console.log(imageDataDisplayP3.colorSpace); // prints "display-p3"
console.log(imageDataDisplayP3.colorType); // prints "float32"
imageData.data[0] = 1.0;
imageData.data[1] = 0.0;
imageData.data[2] = 0.0;
imageData.data[3] = 1.0;
context.putImageData(imageDataDisplayP3, 2, 0);

// Read back the result.
let imageDataReadback = context.getImageData(0, 0, 3, 1);
console.log(imageDataReadback.colorSpace); // prints "display-p3"
console.log(imageDataReadback.colorType); // prints "unorm8"
console.log(imageDataReadback.data); // prints [255, 0, 0, 255 ...] three times.
```

### Reading back contents

The following example draws the color `color(display-p3 1 0 0)` and reads it back.

```javascript
let canvas = document.getElementById('MyCanvas');
This specification is consistent with the default treatment of out-of-device-range values in [WebGPU](https://www.w3.org/TR/webgpu/#gpucanvastonemappingmode) and WebGL.
All other tone mapping modes are to be unified between WebGPU, WebGL, and 2D canvas, via updates to the Canvas High Dynamic Range proposal.

let context = canvas.getContext('2d', {colorSpace:"srgb", colorType:"float16"});
console.log(context.getContextAttributes().colorSpace) // Prints "srgb"
console.log(context.getContextAttributes().colorType) // Prints "float16"
### ImageData

// Draw using CSS colors.
context.fillStyle = "color(display-p3 1 0 0)";
context.fillRect(0, 0, 1, 1);
Previous versions of this specification included support for floating-point types in `ImageData`.
This has now been moved to [this separate issue](https://github.com/whatwg/html/issues/10856).

// Read back with default parameters. Note that the resulting values are
// outside of the [0, 1] interval.
let imageDataDefault = context.getImageData(0, 0, 1, 1);
console.log(imageDataDefault.colorSpace); // prints "srgb"
console.log(imageDataDefault.colorType); // prints "float32"
console.log(imageDataDefault.data); // prints [1.0931, -0.2267, -0.1501, 1.0]

// Read back forcing colorType to "unorm8". Results are clamped.
let imageDataUnorm8 = context.getImageData(0, 0, 1, 1, {colorType:"unorm8"});
console.log(imageDataUnorm8.colorSpace); // prints "srgb"
console.log(imageDataUnorm8.colorType); // prints "unorm8"
console.log(imageDataUnorm8.data); // prints [255, 0, 0, 255]

// Read back forcing colorSpace to "display-p3". The result has colorType
// "float32" by default.
let imageDataDisplayP3 = context.getImageData(0, 0, 1, 1, {colorSpace:"display-p3"});
console.log(imageDataDisplayP3.colorSpace); // prints "display-p3"
console.log(imageDataDisplayP3.colorType); // prints "float32"
console.log(imageDataDisplayP3.data); // prints [1.0, 0.0, 0.0, 1.0]
```

### Determining ImageData type

Consider the function `setToGradient` that initializes an `ImageData` to a horizontal gradient.
This shows how to handle different `ImageData` types.

```javascript
function setToGradient(imageData) {
for (var x = 0; x < imageData.width; ++x) {
for (var y = 0; y < imageData.height; ++y) {
let offset = 4 * x * imageData.height;
switch (imageData.colorType) {
case "unorm8":
imageData.data[offset+0] =
imageData.data[offset+1] =
imageData.data[offset+2] = 255 * x / (imageData.width - 1);
imageData.data[offset+3] = 255;
break;
case "float32":
imageData.data[offset+0] =
imageData.data[offset+1] =
imageData.data[offset+2] = x / (imageData.width - 1);
imageData.data[offset+3] = 1.0;
break;
default:
throw("Unexpected ImageData colorType " + imageData.colorType);
}
}
}
}
```
`ImageData` is a source type for WebGL's `texImage2D` and related functions, WebGPU's `GPUCopyExternalImageSourceInfo` and related functions, and the `createImageBitmap` function.
All new types supported by `ImageData` must be supported and tested in the context of all of these APIs.

## Remarks on choices

### Color type versus pixel format

This proposal uses the term "color type" instead of "pixel format" intentionally to indicate just the precision of the color channels of the format, and not their layout.
For example, a user agent may choose BGRA or RGBA as the representation of an 8 bit per channel canvas, and this implementation detail is hidden.
This proposal uses the term "color type" instead of "pixel format" intentionally to indicate just the data type of the color channels of the format, and not their layout.

The following are problems that this choice avoids:
* A user agent may choose "bgra8unorm" or "rgba8unorm" as the representation of an 8 bit per channel canvas.
This this implementation detail is now hidden.
* The incorporation of the `alpha` boolean can be done in many different ways.
E.g, WebGL has `RGB8` and `RGBA8`.
E.g, WebGPU does has an implicit RGBX behavior via `GPUCanvasAlphaMode` of `opaque`.
These details are avoided by hiding this detail.

In the future, it may be that we will want to add a format like WebGPU's `rgb10a2unorm`.
This format is different in that the `CanvasColorType` would need to indicate RGB and A separately.
This could be done by via `{colorType:"unorm10-unorm2alpha"}` or `{colorType:"unorm10", alpha:false}`.

### Choice of `"float16"` for `CanvasColorType`

The ability to texture from and render to 16 bit floating-point is universal among modern GPUs.

### Choice of `Float32Array` for `ImageData`
### Choice of default behavior of `getImageData`

Historically, all calls to `getImageData` have returned an `ImageData` with a `Uint8ClampedArray`.
Changing the default type that is returned by this function will break any software that relies on
that default type, which is currently all software that uses this function.

There is no plan to add floating-point support for `getImageData`.

There does not exist a `Float16Array` type, and so it is not an option.
The only other floating-point type, `Float64Array`, is of unnecessarily high precision.
A new API, `getImageDataAsync`, which takes an already-created `ImageData` as a parameter is being developed.
By its construction, this API does not suffer from any ambiguity about default parameters.

### Pitfall of `Float32Array` for `ImageData`
See [this issue](https://github.com/whatwg/html/issues/10855) for details.

There exists a potential pitfall wherein a naive user of `ImageData` may write values intended for a `Uint8ClampedArray` to a `Float32Array`
E.g, one would write `255` instead of `1.0`.
### Choice of image encoding behaviors

This can be avoided by examining the `colorType` member the `ImageData`.
It is possible to encode pixel values outside of the [0,1] interval using HDR color spaces (e.g, Rec2100 PQ and Rec2100 HLG).

### Choice of defaulting `getImageData` to `"unorm8"`
Do not exploit this capability yet.
Keep the existing behavior of ensuring of having the encoded image have the same color space as the canvas output bitmap.

Historically, all calls to `getImageData` have returned an `ImageData` with a `Uint8ClampedArray`.
Future work that adds HDR color spaces (e.g: `rec2100-linear`) will default to using Rec2100 PQ encoding, preserving this behavior.

Changing the default type that is returned by this function will break any software that relies on
that default type, which is currently all software that uses this function.
### Feature detection

Changing the default type that is returned by this function will significantly hinder the adoption
of `"float16"` canvas, because no application can change the backing of any canvas without first
ensuring that all libraries that it uses have been updated to support all possible return values.
The presence of this feature can be detected by checking for the presence of `colorType` in the `CanvasRenderingContext2DSettings` returned by `getContextAttributes()`.

This method is not present for `OffscreenCanvasRenderingContext2D`, but that is likely [a spec oversight](https://github.com/whatwg/html/issues/10872).