fix: native ansi-errors raised as spark error

[peterxcli]

Which issue does this PR close?

Closes #5072.

Rationale for this change

Several native ANSI decimal paths returned plain Arrow errors. Those errors bypassed Comet's structured Spark error conversion and surfaced as generic exceptions without Spark's error class, SQLSTATE, or query context.

What changes are included in this PR?

• Return typed SparkError variants for wide-decimal overflow, decimal division by zero, integral division overflow, and decimal-to-decimal cast overflow.
• Preserve query context when CheckOverflow is skipped or when an outer Cast evaluates a failing child expression.
• Remove the decimal-cast raw Arrow error accommodation and add Rust and Spark regressions that compare the Spark error class, SQLSTATE, and query context.

How are these changes tested?

• cd native && cargo test -p datafusion-comet-spark-expr returns_spark_error
• cd native && cargo fmt --all -- --check
• make core
• Spark 4.1.2 with JDK 17: focused CometExpressionSuite tests for ANSI decimal division by zero and wide-decimal overflow, plus the CometCastSuite decimal precision/scale cast test.
• ./mvnw spotless:check

[andygrove]

Thanks for taking this on. I traced through why each piece of the fix is needed and the design is coherent. For a / b the context comes from the injected CheckOverflow, whose Origin is inherited through transformUp in DecimalPrecision.promote, and for a div b it comes from the outer Cast, because CometIntegralDivide builds its inner CheckOverflow proto by hand without an expr_id. That explains why both the planner change and the Cast::evaluate change are required. Deleting the castTest special case and still passing the Spark 3.4/3.5 exact-message assertion is good evidence the value formatting is right for casts. Nice touch keeping the value scan on the error path only.

A few things I would like to work through before merge.

The fused rescale path is still a raw Arrow error

DecimalRescaleCheckOverflow still raises ArrowError::ComputeError on both overflow branches when fail_on_error is set, and it takes no query_context at all:

• native/spark-expr/src/math_funcs/internal/decimal_rescale_check.rs:118 ("Decimal overflow during rescale")
• native/spark-expr/src/math_funcs/internal/decimal_rescale_check.rs:138 ("Decimal overflow: value does not fit in precision")

That looks like the same bug class this PR is fixing everywhere else. The reason it matters for this PR in particular is that planner.rs:531 fuses CheckOverflow(Cast(Decimal128 -> Decimal128)) into that expression, and the fusion check runs before the new cast_array arm can ever be reached. So if that shape is reachable, the decimal-to-decimal cast path from #5072 is only half fixed.

Do you know whether that pattern is reachable on Spark 3.4+? DecimalPrecision.promote only ever wraps a BinaryArithmetic, and arithmetic.scala only ever wraps a divide, so I could not convince myself either way. If it is reachable it would be good to cover it here. If it is not, a tracking issue plus a note on the fusion would be great so this does not get lost.

The reported value diverges from Spark for wide-decimal overflow

At native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs:353 the value handed to NumericValueOutOfRange is the i256 intermediate after rescaling to s_out, formatted as plain digits. Spark goes through Decimal.$times, which multiplies with MATH_CONTEXT (MathContext(38, HALF_UP)), and the error carries Decimal.toString of the pre-rounding value.

For the repro in #5072, 11000000000000000000 * 11000000000000000000, I believe that means Spark reports 1.2100000000000000000000000000000000000E+38 and Comet reports 121000000000000000000000000000000000000. Since value is substituted into the error-class template on the JVM side, it ends up in the user-facing message. The new test only compares error class and SQLSTATE, so the difference does not show up.

Could you confirm what Spark actually prints for that query? If they do diverge, a comment at the call site plus a note in the test explaining why we only compare class and SQLSTATE would help the next reader, and a tracking issue for the message text would be good.

Query context is looked up and then dropped where it originates

create_binary_expr resolves query_context at native/core/src/execution/planner.rs:899, but the wide-decimal arm at line 936 and the decimal-divide arm at line 962 both drop it. Only the integer and float arm wraps in CheckedBinaryExpr. So the fix relies on an ancestor CheckOverflow or Cast carrying the same context, which is a coupling that is hard to see from either side.

Would it be cleaner to wrap WideDecimalBinaryExpr and the decimal_div / decimal_integral_div ScalarFunctionExpr in CheckedBinaryExpr right there, the way the integer arm already does? That would also cover the shapes where no outer CheckOverflow gets emitted, for example when serializeDataType returns None in CometDivide.convert.

Related, at native/core/src/execution/planner.rs:556: with that change, is the wrap on CheckOverflow's child still needed? As written it goes on unconditionally, including when query_context is None, and CheckOverflow still holds its own query_context for its own errors, so we end up with two mechanisms for one job. Gating on query_context.is_some() at minimum would keep the extra node out of plans that cannot use it.

Shared error-unwrap helper

The ArrowError::ExternalError to DataFusionError::External unwrap at native/spark-expr/src/math_funcs/div.rs:186 is identical to the one at native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs:284. Dropping it buries the SparkError inside DataFusionError::ArrowError and puts us right back where #5072 started. Would you mind pulling it into a small shared helper next to the other error helpers, with a short comment on why the ExternalError hop through try_binary is necessary? Future kernels that need to raise a SparkError from an Arrow closure would then get it right by default.

format_decimal_str

At native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs:352, passing the digit count as the precision argument is really a way to defeat the truncation that format_decimal_str applies, which took me a while to work out. A comment saying so would help. More generally, now that this helper is shared between conversion_funcs and math_funcs, would it be worth a non-truncating variant, or moving it somewhere neutral? A formatter whose precision parameter silently drops digits is easy to misuse from a call site that does not already know the value fits.

Tests

At native/spark-expr/src/conversion_funcs/cast.rs:342, a couple of cases I would like to see covered. Negative values exercise the sign handling in format_decimal_str. A scale-up overflow goes through make_upscaler rather than make_downscaler, so it is a different code path inside rescale_decimal. It would also be reassuring to have a case where the offending value is not the first row and there are nulls ahead of it, to pin the flatten().find() behavior.

At native/spark-expr/src/math_funcs/wide_decimal_binary_expr.rs:524, would it be worth a Multiply case alongside the Add one? Multiply with s_out < s1 + s2 goes through div_round_half_up before the overflow check, so the value reported in the error comes out of a different branch than the one this test covers.

At spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:52, any reason for comparing getClass.getName against the string rather than pattern matching on SparkArithmeticException directly? The string form will not match a subclass and will fail quietly if the class ever moves. If it is working around a cross-version compile problem, a comment saying which version would be helpful.

One note on test placement, since I checked: CometSqlFileTestSuite's expect_error only does a substring match on the message and cannot assert error class, SQLSTATE, or query context, so the Scala tests are the right call here.

Cast::evaluate

At native/spark-expr/src/conversion_funcs/cast.rs:786, routing the child's error through the context-wrapping block is what makes the a div b test work, since CometIntegralDivide registers the IntegralDivide's context on the outer Cast proto. That is worth a comment, because in the general case this attaches the cast's own fragment to an error raised by a child, where Spark would report the child's fragment. It only bites when the child raises a bare SparkError with no context of its own, which is a case where today we report nothing at all, so it is an improvement either way. A note saying it is deliberate would stop someone reading it as a bug later.

[peterxcli]

1: The fused rescale path is still a raw Arrow error

Changed DecimalRescaleCheckOverflow to raise SparkError::NumericValueOutOfRange, unwrap Arrow external errors, and propagate query context. Large scale deltas now preserve zero/null behavior and return typed ANSI errors. Added a note that current Spark 3.4–4.2 serialization does not reach this fusion.

2: The reported value diverges from Spark for wide-decimal overflow

Confirmed the message-value difference and documented it at the Rust call site and in the Scala test. The test intentionally compares error class, SQLSTATE, and query context rather than the differing value parameter.
Opened #5211 with the example query and corresponding Spark/Comet messages. Linked it from the Rust call site and Scala regression-test comment.

3: Query context is looked up and then dropped where it originates

Wide-decimal and decimal-division expressions are now wrapped directly when query context exists. The ancestor wrapper remains as a fallback for generated child protos without an expr_id, but context-less and duplicate wrappers are avoided.

4: Shared error-unwrap helper

Added a shared Arrow external-error unwrapping helper and reused it in decimal division, wide-decimal arithmetic, and fused decimal rescaling.

5: format_decimal_str

Removed Comet’s copied formatter and reused Arrow’s public format_decimal_str. Added comments explaining why the actual digit count is passed for already-overflowing values.

6: Tests

Added coverage for negative decimal casts, scale-up overflow, null and valid rows before the offending value, rounded multiplication overflow, fused array/scalar overflow, and large scale deltas.

The SparkArithmeticException class-name match remains because that type is inaccessible outside org.apache.spark in Spark 3.4. Added a comment documenting this cross-version constraint.

7: Cast::evaluate

Added a comment explaining that child errors deliberately inherit the outer Cast query context because CometIntegralDivide creates its inner CheckOverflow without an expression ID.