`arrays_except` type mismatch

[comphead]

Describe the bug

- sql-file: expressions/array/array_except.sql [parquet.enable.dictionary=true] *** FAILED *** (733 milliseconds)
  java.lang.RuntimeException: Error executing SQL 'SELECT array_except(array(1, 2, 3), array(2, 3, 4)), array_except(array(1, 2), array()), array_except(array(), array(1)), array_except(cast(NULL as array<int>), array(1))'. Reason: 'Job aborted due to stage failure: Task 0 in stage 23.0 failed 1 times, most recent failure: Lost task 0.0 in stage 23.0 (TID 103) (192.168.4.142 executor driver): org.apache.comet.CometNativeException: array_except received incompatible types: List(Int32), List(non-null Int32)

This happens because exhaustive datatype check in datafusion in check_datatypes() function

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[n0r0shi]

For this, I think there are 2 approaches. One is to fix upstream: make check_datatypes() ignore inner field nullability. Another is to fix in Comet: normalize inner nullability before calling array_except. Avoids upstream change but adds overhead.

I'd lean toward (1), but I'd like to confirm how others think @comphead

[comphead]

I tried to replace .equals_datatype to .contains which is more nullability friendly, however it would still doesn't work this scenario.

.contains ignores nullability if the primary datatype is nullable, but let data_type = args[0].data_type(); doesn't guarantee exactly this datatype would be nullable.

I feel we can tweak check_datatype and if incoming datatypes are lists then take first NULLABLE datatype instead of just first