Skip to content

[spark] Reject recursive Paimon view definitions - #10231

Open
Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:xinyu/fix-recursive-paimon-views
Open

Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:xinyu/fix-recursive-paimon-views

Conversation

@Zouxxyy

@Zouxxyy Zouxxyy commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Purpose

CREATE OR REPLACE VIEW v AS SELECT * FROM v can analyze the old definition and then persist a recursive Paimon view. Reading the stored definition subsequently recurses until StackOverflowError.

Validate the original SQL before replacing metadata, preserving the existing definition when a cycle is found. Also reject legacy persisted cycles during resolution and report their dependency path. Paimon expands views without retaining Spark's logical View nodes, so the checker follows stored SQL dependencies, including nested subqueries and CTEs, and respects catalog identifier case sensitivity.

The guard points follow Spark's ViewHelper and ViewResolution. A separate internal checker enforces spark.sql.view.maxNestedViewDepth and reuses validated subtree depths only within one resolver traversal. This avoids repeatedly scanning a nested view chain while still checking shared dependencies reached through deeper paths.

Tests

  • PaimonViewTest: 20 tests passed on each of Spark 3.5 / Java 8, Spark 4.0 / Java 17, and Spark 4.1 / Java 17.
  • Covers direct and indirect cycles, legacy persisted cycles and recovery, EXISTS/IN/scalar subqueries, CTE dependencies, mixed-case identifiers, valid CTE shadowing, and shared dependencies.
  • Regression coverage bounds catalog lookups for a nested view chain, detects metadata changes on the next resolver invocation, and checks nesting limits through shared dependencies.
  • Final Spark 3 and Spark 4 Maven test builds passed with Checkstyle and Spotless enabled.
  • For the local Spark 4.1 run, only the generated test Hive configuration used port 19092 because port 9092 was occupied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant