[SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE

### What changes were proposed in this pull request?

Fix correctness with UNION/EXCEPT/INTERSECT inside a view or `EXECUTE IMMEDIATE`.

In the following examples the SQL Parser considers UNION/EXCEPT/INTERSECT keywords as aliases and drops the rest of the query:

```
spark.sql("CREATE OR REPLACE VIEW v1 AS SELECT 1 AS col1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4")
spark.sql("SELECT * FROM v1").show()
spark.sql("SELECT * FROM v1").queryExecution.analyzed

spark.sql("CREATE OR REPLACE VIEW v1 AS SELECT 1 AS col1 EXCEPT SELECT 2 EXCEPT SELECT 1 EXCEPT SELECT 2")
spark.sql("SELECT * FROM v1").show()
spark.sql("SELECT * FROM v1").queryExecution.analyzed

spark.sql("CREATE OR REPLACE VIEW t1 AS SELECT 1 AS col1 INTERSECT SELECT 1 INTERSECT SELECT 2 INTERSECT SELECT 2")
spark.sql("SELECT * FROM v1").show()
spark.sql("SELECT * FROM v1").queryExecution.analyzed

spark.sql("DECLARE v INT")
spark.sql("EXECUTE IMMEDIATE 'SELECT 1 UNION SELECT 2 UNION SELECT 3' INTO v")
spark.sql("EXECUTE IMMEDIATE 'SELECT 1 UNION SELECT 2 UNION SELECT 3' INTO v").queryExecution.analyzed
spark.sql("SELECT v").show()
```

![image](https://github.com/user-attachments/assets/ef726178-2375-4ebc-a7e3-88f1991d1016)
![image](https://github.com/user-attachments/assets/50b4b7ba-bc7d-4fc1-a921-f4cbfcab79a3)
![image](https://github.com/user-attachments/assets/85b65325-5dd9-4d74-b46d-8ea203ce1039)
![image](https://github.com/user-attachments/assets/c53c5e02-18c6-4e30-b834-94af619190c5)

There's no correctness issue associated with regular queries (without the `VIEW` or `EXECUTE IMMEDIATE`). Apparently that's because we use `ParserInterface.parsePlan` (`singleStatement` term in Spark SQL grammar) for [regular queries](https://github.com/apache/spark/blob/b968ce1d3ac1b72019b30bf3d4e11d9574ba1205/sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala#L490) and `ParserInterface.parseQuery` (`query` term in Spark SQL grammar) for [view bodies](https://github.com/apache/spark/blob/b968ce1d3ac1b72019b30bf3d4e11d9574ba1205/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L986) and [EXECUTE IMMEDIATE with INTO](https://github.com/apache/spark/blob/b968ce1d3ac1b72019b30bf3d4e11d9574ba1205/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/executeImmediate.scala#L167). The difference is that `singleStatement` [ends in EOF](https://github.com/apache/spark/blob/b968ce1d3ac1b72019b30bf3d4e11d9574ba1205/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4#L144).

Not sure what's the actual root cause, because I don't know much about the SQL Parser.

### Why are the changes needed?

Correctness issue fix.

### Does this PR introduce _any_ user-facing change?

Yes, the results of queries on top of aforementioned views are gonna be correct.

### How was this patch tested?

New `view-correctness` suite.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#49937 from vladimirg-db/vladimirg-db/fix-views-with-trivial-unions-2.

Authored-by: Vladimir Golubev <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: vladimirg-db

sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

@@ -475,6 +475,10 @@ commentSpec
     : COMMENT stringLit
     ;
 
+singleQuery
+    : query EOF
+    ;
+
 query
     : ctes? queryTerm queryOrganization
     ;

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala

@@ -24,6 +24,7 @@ import org.apache.spark.sql.catalyst.parser.ParserUtils.withOrigin
 import org.apache.spark.sql.catalyst.plans.logical.{CompoundPlanStatement, LogicalPlan}
 import org.apache.spark.sql.catalyst.trees.Origin
 import org.apache.spark.sql.errors.QueryParsingErrors
+import org.apache.spark.sql.internal.SQLConf
 
 /**
  * Base class for all ANTLR4 [[ParserInterface]] implementations.
@@ -72,9 +73,18 @@ abstract class AbstractSqlParser extends AbstractParser with ParserInterface {
   /** Creates LogicalPlan for a given SQL string of query. */
   override def parseQuery(sqlText: String): LogicalPlan =
     parse(sqlText) { parser =>
-      val ctx = parser.query()
-      withErrorHandling(ctx, Some(sqlText)) {
-        astBuilder.visitQuery(ctx)
+      if (!SQLConf.get.getConf(SQLConf.LEGACY_PARSE_QUERY_WITHOUT_EOF)) {
+        val ctx = parser.singleQuery()
+
+        withErrorHandling(ctx, Some(sqlText)) {
+          astBuilder.visitSingleQuery(ctx)
+        }
+      } else {
+        val ctx = parser.query()
+
+        withErrorHandling(ctx, Some(sqlText)) {
+          astBuilder.visitQuery(ctx)
+        }
       }
     }
 

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

@@ -670,6 +670,10 @@ class AstBuilder extends DataTypeAstBuilder
    * ******************************************************************************************** */
   protected def plan(tree: ParserRuleContext): LogicalPlan = typedVisit(tree)
 
+  override def visitSingleQuery(ctx: SingleQueryContext): LogicalPlan = withOrigin(ctx) {
+    visitQuery(ctx.query())
+  }
+
   /**
    * Create a top-level plan with Common Table Expressions.
    */

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -5520,6 +5520,18 @@ object SQLConf {
     .booleanConf
     .createWithDefault(true)
 
+  val LEGACY_PARSE_QUERY_WITHOUT_EOF = buildConf("spark.sql.legacy.parseQueryWithoutEof")
+    .internal()
+    .doc(
+      "When set to true, ParserInterface#parseQuery(...) is going to use base `query` grammar " +
+      "term without EOF resulting in some queries (like `SELECT 1 UNION SELECT 2`) to be parsed " +
+      "incorrectly - `UNION` will be treated as an alias, and the rest of SQL input will be " +
+      "thrown away."
+    )
+    .version("4.0.0")
+    .booleanConf
+    .createWithDefault(false)
+
   /**
    * Holds information about keys that have been deprecated.
    *