Accumulation precision is unspecified for matmul/gemm and reduction operations

[mtavenrath]

Accumulation precision for operations with internal summation

Problem

The WebNN spec does not say anything about what precision intermediate accumulations should use. This is most impactful for matmul and gemm, but applies to any operation that internally sums many values (reduceSum, conv2d, the dot products inside gru/lstm, etc.).

In practice, the accumulation precision is the single biggest factor determining numerical accuracy of a matrix multiply. And different backends make very different choices here, silently.

Why this matters in practice

Take a float16 matmul with K=1024 (a very normal dimension for transformer models):

• float32 accumulation: relative error bounded by roughly K × ε₃₂ ≈ 1024 × 6×10⁻⁸ ≈ 6×10⁻⁵. Fine.
• float16 accumulation: relative error bounded by roughly K × ε₁₆ ≈ 1024 × 10⁻³ ≈ 1.0. The error can be as large as the result itself.

This is not a theoretical corner case. It is the difference between a model working and producing garbage.

A minimal example: dot product of two float16 vectors of length 1024, all values 1.0:

a = [1.0, 1.0, ..., 1.0]  // 1024 elements, float16
b = [1.0, 1.0, ..., 1.0]  // 1024 elements, float16

Expected: dot(a, b) = 1024.0

With float32 accumulation you get exactly 1024.0. With float16 accumulation, once the running sum exceeds 2048, adding 1.0 starts getting rounded away. For real-world non-uniform data, errors compound much worse.

What backends actually do

Some backends use float32 accumulation for float16 inputs:

• NVIDIA cuBLAS: Default compute type for float16 inputs is CUBLAS_COMPUTE_32F (documented).
• Intel oneDNN: Default accumulation mode is strict, which means float32 for float ops (documented). Their data types guide explicitly states that intermediate computations use higher precision to avoid overflow (documented).
• XNNPACK: Operates in float32 internally; float16 is just a storage format.

Others may use float16 accumulation for throughput, especially on mobile/edge accelerators. The developer cannot tell what they are getting from the WebNN API, and cannot request anything specific.

Possible approaches

Approach A: Explicit upcast

In ONNX and TensorRT, accumulation precision is controlled by casting inputs to float32, performing the operation, and casting back:

const a_f32 = builder.cast(a_f16, "float32");
const b_f32 = builder.cast(b_f16, "float32");
const c_f32 = builder.matmul(a_f32, b_f32);
const c_f16 = builder.cast(c_f32, "float16");

The backend (or a graph optimizer) recognizes the cast-compute-cast pattern and fuses it into an efficient kernel that reads float16 inputs while accumulating in float32 internally.

The spec guarantees only that the accumulator precision is at least the input type. When no casts are present, a float16 matmul accumulates in at least float16 — the backend may silently use higher precision. When the developer needs a specific accumulator precision, they express it through explicit casts.

Advantages:

• No new API surface. Uses existing cast + operator primitives.
• Familiar pattern from ONNX and TensorRT ecosystems.
• Least intrusive spec change.

Disadvantages:

• Relies on the backend being smart enough to fuse the pattern. A naive backend might actually materialize the float32 intermediates, losing the memory/bandwidth benefit of float16.
• Verbose when many operations need the same accumulation policy.

Approach B: Explicit accumulator type per operation

Add an optional accumulatorType field to operations that have internal accumulation:

partial dictionary MLMatmulOptions {
  MLOperandDataType accumulatorType;
};

// Float32 accumulation for attention matmul
builder.matmul(q_k, v, { accumulatorType: "float32" });

// Don't care, go fast (backend picks freely)
builder.matmul(a, b);

The spec guarantees only that the accumulator precision is at least the input type. When accumulatorType is omitted, the backend may use any accumulator precision equal to or higher than the input type — a float16 matmul might accumulate in float16 or float32, at the backend's discretion. When accumulatorType is specified, the backend must use at least that precision. If it cannot, it rejects the graph at build time rather than silently falling back.

This matches the reality of the underlying APIs. In Vulkan cooperative matrices, WebGPU subgroup matrices, and D3D12 linalg, the accumulator type is an explicit parameter — and all three guarantee at least the requested precision. When a lower accumulator type is requested (e.g. f16), the hardware may internally accumulate at higher precision and round the result back — D3D12 makes this visible via the EMULATED_OUTPUTS flag; Vulkan and WebGPU leave it implicit. But when a higher type is requested (e.g. f32), it accumulates in that type.

Advantages:

• Simple and explicit. One optional field, one clear meaning.
• Easy to implement in backends that already have a compute-type concept.
• No ambiguity — the developer names exactly what they want.

Disadvantages:

• New API surface on every applicable operation.
• Ties the set of valid accumulator types to MLOperandDataType. If a future hardware-internal format is added, the enum must grow.

Discoverability

The support limits could report what the backend provides:

const limits = context.opSupportLimits().matmul;
console.log(limits.accumulatorType);
// e.g. "float32"

Integration with model runtimes (ORT-Web, LiteRT.js)

Neither ONNX nor TFLite specify accumulator precision in their model formats today. The decision falls to whoever sits between the model file and the hardware.

For Approach A, runtimes insert casts when translating the model graph to WebNN. For Approach B, runtimes set accumulatorType per-op based on a session-level policy:

// ORT-Web example
const session = await ort.InferenceSession.create('model.onnx', {
  executionProviders: [{
    name: 'webnn',
    accumulationPrecision: 'high',  // maps to accumulatorType: "float32" on all ops
  }],
});

No model format changes are required.

Scope

This applies to any operation with internal summation:

• matmul / gemm
• conv2d / convTranspose2d
• reduceSum / reduceMean / reduceLogSumExp
• gru / lstm
• linear (if added)

What other APIs do for comparison

API	Mechanism	Accumulator handling
Vulkan cooperative matrices	Accumulator type is an explicit template parameter (CType/ResultType) per operation. Hardware queries which (AType, BType, CType, ResultType) combinations are supported.	Fully explicit; the developer picks the accumulator type from the supported list.
cuBLAS	cublasComputeType_t enum passed at call time.	Named enum: CUBLAS_COMPUTE_32F, CUBLAS_COMPUTE_16F, etc.
TensorRT	Express via casts in the graph; runtime fuses the pattern.	No new API surface; relies on optimizer pattern matching.
oneDNN	dnnl::accumulation_mode enum (strict = fp32, any = fast).	Combines intent-based with type-based.
Metal Performance Shaders	allowReducedPrecision boolean flag.	Binary opt-in; no granularity.
WebGPU subgroup matrices	resultComponentType in GPUSubgroupMatrixConfig is an explicit, separate type from componentType. The device enumerates supported (componentType, resultComponentType, M, N, K) tuples; the application picks one.	Explicit per-configuration. The result type determines accumulation precision (e.g. componentType: "f16", resultComponentType: "f32" means f16 inputs with f32 accumulation).
D3D12 linalg (SM 6.10)	AccumulatorComponentType is an explicit, separate field in D3D12_LINEAR_ALGEBRA_WAVE_MATRIX_MULTIPLY_INPUTS. The application queries whether a specific (MatrixAComponentType, MatrixBComponentType, AccumulatorComponentType) combination is supported at a given shape.	Explicit per-operation query. The EMULATED_OUTPUTS flag signals when the hardware internally uses higher precision than the requested accumulator type. Tier 1 requires f16×f16→f16 and explicitly permits backends to accumulate at higher precision and convert.

Both WebGPU and D3D12 treat the accumulator as a first-class, queryable parameter — the application names a specific type, and the backend reports whether it supports that combination. Neither API is silent about accumulation; both make it explicit and discoverable.

WebNN being silent on accumulation means the same model can produce different results across backends and the developer has no recourse.

References

• Vulkan VkCooperativeMatrixPropertiesKHR
• cuBLAS cublasComputeType_t
• TensorRT Precision Control
• TensorRT Accuracy Considerations
• oneDNN accumulation_mode
• oneDNN Numerical Behavior
• WebGPU Subgroup Matrix proposal
• D3D12 Linear Algebra Runtime Feature Support

[fdwr]

Both WebGPU and D3D12 treat the accumulator as a first-class, queryable parameter ... What other APIs do for comparison

Another entry for your table is DirectML's DML_EXECUTION_FLAG_ALLOW_HALF_PRECISION_COMPUTATION, which allowed float16 accumulation with float16 inputs, whereas default accumulator computation was float32 for float16 inputs (casting back to float16 for output).