Skip to content

docs: add a Frequently Asked Questions page - #5196

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:docs-faq
Draft

docs: add a Frequently Asked Questions page#5196
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:docs-faq

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #.

Rationale for this change

We get the same handful of user questions repeatedly, and while most of the answers already exist in the docs, they are spread across the tuning guide, the plan guide, the compatibility guide, the roadmap, and the Gluten comparison. A new user with a question phrased in their own words has no entry point to them.

Looking through GitHub discussions and issues for the recurring themes, these came up most:

Source Already answered somewhere?
#3199, #1084 — benchmarked and Spark was faster / no speedup scattered across tuning + plan guides
#5001 — native OOM on large joins/distinct where Spark succeeds partly in the tuning guide's memory section
#1018, #4648, #174 — plans for Delta Lake / the data lake ecosystem roadmap, but that lives under the contributor guide
#3043 — Spark 3.3 / JDK 8 support implied by the supported-versions table, never stated as a "no"
#1196 — does Comet use SIMD nowhere
#994 — comparison with Gluten yes, about/gluten_comparison.md
#1014 — jars for Apple Silicon yes, installation guide

What changes are included in this PR?

Adds docs/source/faq.md with 15 questions in four sections (Getting Started, Performance, Compatibility and Support, Project and Community), and links it from the top-level toctree in index.md and from the homepage "Getting Started" blurb.

Two things worth calling out for review:

The page is a routing layer, not a new source of truth. Every answer is a few sentences that ends in a link into the relevant guide. Nothing is documented only here. That is the property that stops the page drifting out of sync with the versioned user guide, and I'd suggest we keep it as the rule for future additions.

The page is unversioned, and links into user-guide/latest. Several answers are roadmap- and project-level ("do you plan to support Delta Lake?", "what's coming next?"), which would go stale if frozen into a per-release snapshot, and versioning the page would mean maintaining N copies of it. Linking to latest follows what the homepage CTAs and gluten_comparison.md already do — but note latest is the development snapshot rather than the current stable release, so the FAQ points users slightly ahead of what most of them are running. I followed the existing convention rather than diverging, and said so in the page intro. Happy to change it if we'd rather unversioned pages targeted stable, though that feels like a separate site-wide decision.

On the Spark/JDK question specifically: the JDK 11 and Spark 3.4 deprecation is already documented in three places (installation.md, compatibility/spark-versions.md, and versioning_policy.md), so the FAQ restates it as a warning admonition and links out rather than becoming a fourth place to keep in sync.

How are these changes tested?

Docs only, no code changes.

  • Verified all 14 relative file links resolve and all 10 cross-file heading anchors match a real heading in the target file.
  • npx prettier "**/*.md" reports both files clean.

I was not able to build the docs locally — docs/requirements.txt fails to install in my environment — so the toctree entry is unexercised until CI builds the site. Worth a look at the rendered output before this comes out of draft.

Adds an unversioned FAQ at docs/source/faq.md covering the questions that
come up most often in GitHub discussions and issues, and links it from the
top-level toctree and the homepage "Getting Started" section.

The page is deliberately a routing layer rather than a new source of truth.
Most of these answers already exist somewhere in the docs -- the tuning
guide, the plan guide, the compatibility guide, the roadmap, the Gluten
comparison -- but a new user with a question phrased in their own words has
no entry point to them. Every answer here is a few sentences that ends in a
link into the relevant guide, so nothing is duplicated and the page does not
drift out of sync with the versioned user guide.

Questions were chosen from recurring discussions and issues, including:

- Why is my query no faster, or slower, with Comet? (apache#3199, apache#1084)
- Why does Comet OOM where Spark succeeded? (apache#5001)
- Does Comet support Delta Lake, Hudi, or Paimon? (apache#1018, apache#4648, apache#174)
- Can I use Comet with Spark 3.3 or Java 8? (apache#3043)
- Does Comet use SIMD? (apache#1196)
- How does Comet compare to Gluten? (apache#994)

The page is unversioned because several answers are roadmap- and
project-level, which would go stale if frozen into a per-release snapshot
and would otherwise need maintaining in every versioned copy. Links point
into user-guide/latest, matching the existing convention on the homepage
and the Gluten comparison page.
Comment thread docs/source/faq.md
executor logs. If it is absent, the native library never loaded and nothing is being accelerated.
2. Read `spark.conf.get("spark.comet.version")` to confirm which build is loaded.
3. Run `EXPLAIN FORMATTED` on the query and look for `Comet*` operators in the plan. Anything without
the `Comet` prefix is running on unmodified Spark.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice having an example that shows a plan with fallbacks

val plan = df.queryExecution.executedPlan
val info = new org.apache.comet.ExtendedExplainInfo()
println(info.generateExtendedInfo(plan))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants