Skip to content

[SPARK-56029] Optional sub error conditions#54861

Open
srielau wants to merge 7 commits intoapache:masterfrom
srielau:SPARK-56029-optional-error-conditions
Open

[SPARK-56029] Optional sub error conditions#54861
srielau wants to merge 7 commits intoapache:masterfrom
srielau:SPARK-56029-optional-error-conditions

Conversation

@srielau
Copy link
Contributor

@srielau srielau commented Mar 17, 2026

What changes were proposed in this pull request?

We make raising of error sub conditions and checking for them optional.
SPARK-56029-optional-error-subconditions.plan.md

Why are the changes needed?

This allows conversion of error conditions without subconditions to ones with possible without churning and causing incompatible changes

Does this PR introduce any user-facing change?

Yes, checkError() is more tolerant

How was this patch tested?

Added error condiiton unit tests

Was this patch authored or co-authored using generative AI tooling?

Claude Opus 4.6 high

@srielau srielau force-pushed the SPARK-56029-optional-error-conditions branch from e94d5df to e328f1f Compare March 18, 2026 02:40
val errorInfo = errorInfoMap.getOrElse(
mainErrorClass,
throw SparkException.internalError(s"Cannot find main error class '$errorClass'"))
assert(errorInfo.subClass.isDefined == subErrorClass.isDefined)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while I understand the motivation, I think this might be too lenient if left just like this. it would be nice to have an "opt-in" mechanism, so we have some kind of an indication that this is a desired behavior for a specific error condition. otherwise, we are very silently converting all error conditions to a new behavior - over time this might lead to subclasses being effectively unused, since the path of least resistance is always the main condition.

I think it's not too complex to mitigate though. We can make an opt-in mechanism in error-conditions.json, something like:

"TABLE_OR_VIEW_NOT_FOUND": {                                                                                                                                      
    "message": ["..."],                                                                                                                                             
    "subClassOptional": true,                                                                                                                                       
    "subClass": {                                                                                                                                                   
      "PATH": { "message": ["..."] }                                                                                                                                
    }                                                                                                                                                               
  }

And then use subClassOptional in ErrorClassesJSONReader to relax the condition when the value is set to true.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to to that. Let's see what other think. Note that this is NOT what we do for SQL Scripting handlers though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants