docs: add a Frequently Asked Questions page - #5196
Draft
andygrove wants to merge 1 commit into
Draft
Conversation
Adds an unversioned FAQ at docs/source/faq.md covering the questions that come up most often in GitHub discussions and issues, and links it from the top-level toctree and the homepage "Getting Started" section. The page is deliberately a routing layer rather than a new source of truth. Most of these answers already exist somewhere in the docs -- the tuning guide, the plan guide, the compatibility guide, the roadmap, the Gluten comparison -- but a new user with a question phrased in their own words has no entry point to them. Every answer here is a few sentences that ends in a link into the relevant guide, so nothing is duplicated and the page does not drift out of sync with the versioned user guide. Questions were chosen from recurring discussions and issues, including: - Why is my query no faster, or slower, with Comet? (apache#3199, apache#1084) - Why does Comet OOM where Spark succeeded? (apache#5001) - Does Comet support Delta Lake, Hudi, or Paimon? (apache#1018, apache#4648, apache#174) - Can I use Comet with Spark 3.3 or Java 8? (apache#3043) - Does Comet use SIMD? (apache#1196) - How does Comet compare to Gluten? (apache#994) The page is unversioned because several answers are roadmap- and project-level, which would go stale if frozen into a per-release snapshot and would otherwise need maintaining in every versioned copy. Links point into user-guide/latest, matching the existing convention on the homepage and the Gluten comparison page.
comphead
reviewed
Aug 1, 2026
| executor logs. If it is absent, the native library never loaded and nothing is being accelerated. | ||
| 2. Read `spark.conf.get("spark.comet.version")` to confirm which build is loaded. | ||
| 3. Run `EXPLAIN FORMATTED` on the query and look for `Comet*` operators in the plan. Anything without | ||
| the `Comet` prefix is running on unmodified Spark. |
Contributor
There was a problem hiding this comment.
would be nice having an example that shows a plan with fallbacks
val plan = df.queryExecution.executedPlan
val info = new org.apache.comet.ExtendedExplainInfo()
println(info.generateExtendedInfo(plan))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
We get the same handful of user questions repeatedly, and while most of the answers already exist in the docs, they are spread across the tuning guide, the plan guide, the compatibility guide, the roadmap, and the Gluten comparison. A new user with a question phrased in their own words has no entry point to them.
Looking through GitHub discussions and issues for the recurring themes, these came up most:
about/gluten_comparison.mdWhat changes are included in this PR?
Adds
docs/source/faq.mdwith 15 questions in four sections (Getting Started, Performance, Compatibility and Support, Project and Community), and links it from the top-level toctree inindex.mdand from the homepage "Getting Started" blurb.Two things worth calling out for review:
The page is a routing layer, not a new source of truth. Every answer is a few sentences that ends in a link into the relevant guide. Nothing is documented only here. That is the property that stops the page drifting out of sync with the versioned user guide, and I'd suggest we keep it as the rule for future additions.
The page is unversioned, and links into
user-guide/latest. Several answers are roadmap- and project-level ("do you plan to support Delta Lake?", "what's coming next?"), which would go stale if frozen into a per-release snapshot, and versioning the page would mean maintaining N copies of it. Linking tolatestfollows what the homepage CTAs andgluten_comparison.mdalready do — but notelatestis the development snapshot rather than the current stable release, so the FAQ points users slightly ahead of what most of them are running. I followed the existing convention rather than diverging, and said so in the page intro. Happy to change it if we'd rather unversioned pages targeted stable, though that feels like a separate site-wide decision.On the Spark/JDK question specifically: the JDK 11 and Spark 3.4 deprecation is already documented in three places (
installation.md,compatibility/spark-versions.md, andversioning_policy.md), so the FAQ restates it as a warning admonition and links out rather than becoming a fourth place to keep in sync.How are these changes tested?
Docs only, no code changes.
npx prettier "**/*.md"reports both files clean.I was not able to build the docs locally —
docs/requirements.txtfails to install in my environment — so the toctree entry is unexercised until CI builds the site. Worth a look at the rendered output before this comes out of draft.