[CELEBORN-2415] Add Celeborn Spark UI extension by yew1eb · Pull Request #3793 · apache/celeborn

yew1eb · 2026-08-10T10:17:44Z

What changes were proposed in this pull request?

Add a new module celeborn-client-spark-3-ui — a Spark UI extension that renders a "Celeborn" tab in both the live Spark UI and the History Server (enabled via spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin, supports Spark 3.x and 4.x).

This is an initial version that lays the foundation; it intentionally starts minimal. The page shows:

Summary — shuffle write/read bytes, time, speed, and shuffle-vs-task duration ratio, aggregated from Spark's built-in TaskMetrics
Celeborn Properties — a collapsible table of spark.celeborn.* configs

The skeleton is: CelebornPlugin (live UI) / CelebornHistoryServerPlugin (SHS, via AppHistoryServerPlugin SPI) → listener aggregates into Spark KVStore → CelebornStatusStore → CelebornUITab renders; a javax/jakarta servlet bridge covers Spark 3.x and 4.x. This plugin/store/render pipeline is designed so that fine-grained Celeborn client metrics can be added as collapsible sections in follow-up PRs:

Shuffle Write Times — serialize / copy / queue wait / compress / queue stall / inflight wait / drain wait / max push RTT / slow push
Shuffle Read Times — chunk wait / decompress / retry wait / max chunk RTT / slow chunk
Shuffle Write Servers — per-worker push bytes, push count, RTT, soft/hard splits, congested counts, last failure reason
Shuffle Read Servers — per-worker chunk count, read bytes, total/max RTT
Shuffle Assignments — partition-to-worker slot assignments

Why are the changes needed?

Spark UI already shows stage/task-level shuffle metrics, but nothing Celeborn-specific: which workers served the data, push/chunk latencies, congestion and split events are invisible there, while Celeborn's own metrics sit in cluster-level Grafana dashboards that cannot be attributed back to a specific application. This tab closes that gap by putting per-application Celeborn shuffle information inside the Spark UI and History Server, alongside the job's stages and tasks.

Does this PR introduce any user-facing change?

Yes.
Setting spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin (with the extension jar on the classpath) adds a "Celeborn" tab to the Spark UI and History Server. Disabled by default; no effect otherwise.

Does this PR resolve a correctness bug?

No.

How was this patch tested?

Manual end-to-end verification:

yew1eb · 2026-08-11T03:56:10Z

Ping @pan3793 @cxzl25 @RexXiong @SteNicholas @zaynt4606

Copilot

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds a new Spark UI / History Server extension module that surfaces Celeborn-related shuffle summary metrics and spark.celeborn.* configuration in a dedicated “Celeborn” tab.

Changes:

Introduces new module client-spark/spark-3-ui with Spark plugin + History Server plugin, status-store access, and UI rendering.
Adds a Spark 3 vs Spark 4 servlet (javax vs jakarta) bridge via profile-selected source roots.
Wires the new UI module into root build and shaded Spark client artifacts.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pom.xml	Adds the new UI module to Spark build profiles and introduces `servlet.source.dir` to select Spark3/Spark4 servlet bridge sources.
client-spark/spark-3-ui/pom.xml	New module POM with Spark dependency + servlet APIs and build-helper source-root selection.
client-spark/spark-4-shaded/pom.xml	Includes the new UI artifact in the Spark 4 shaded assembly dependencies.
client-spark/spark-3-shaded/pom.xml	Includes the new UI artifact in the Spark 3 shaded assembly dependencies.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornPlugin.scala	Adds SparkPlugin entrypoint for live UI, registers listener and attaches tab.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornHistoryServerPlugin.scala	Adds SHS plugin SPI implementation to replay metrics and attach the tab.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornListener.scala	Listener aggregates task metrics and persists rollups + properties into KVStore.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornStatusStore.scala	Status-store accessors for aggregated metrics and celeborn properties.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/ui/CelebornUITab.scala	Defines the “Celeborn” tab and registers the page.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/ui/CelebornShufflePage.scala	Renders summary metrics + collapsible properties table.
client-spark/spark-3-ui/src/main/scala-spark3/org/apache/spark/shuffle/celeborn/ui/SparkServletBridge.scala	Spark 3 servlet type alias (`javax.servlet`).
client-spark/spark-3-ui/src/main/scala-spark4/org/apache/spark/shuffle/celeborn/ui/SparkServletBridge.scala	Spark 4 servlet type alias (`jakarta.servlet`).
client-spark/spark-3-ui/src/main/resources/META-INF/services/org.apache.spark.status.AppHistoryServerPlugin	Registers the SHS plugin via SPI.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

yew1eb · 2026-08-18T11:52:52Z

+private[celeborn] class CelebornPropertiesUIData(
+    val info: Seq[(String, String)]) {
+
+  @JsonIgnore
+  @KVIndex
+  def id: String = classOf[CelebornPropertiesUIData].getName
+}


Not necessary. Verified end-to-end that the properties table renders correctly in both live UI and SHS replay. Spark's KVStore uses Jackson with the Scala

yew1eb · 2026-08-18T11:55:33Z

+    if (statusStore.hasData()) {
+      new CelebornUITab(statusStore, ui)
+    }


Fair point on the live/SHS inconsistency. Fixed by attaching the tab unconditionally in SHS (matching the live UI, which always attaches once spark.plugins=CelebornPlugin is set) and removing the now-unused hasData().

yew1eb · 2026-08-18T11:56:06Z

+  private val totalTaskDurationMs = new AtomicLong(0L)
+
+  private val lastUpdateTimestamp = new AtomicLong(-1L)
+  private val updateIntervalMillis = 5000L


Reasonable for later, but intentionally a fixed default for the MVP. We can expose spark.celeborn.ui.flushIntervalMs in a follow-up once we see real cadence
needs.

yew1eb · 2026-08-18T11:57:00Z

+    val last = lastUpdateTimestamp.get()
+    if (!force && (last != -1L && (now - last) < updateIntervalMillis)) {
+      return
+    }
+    if (lastUpdateTimestamp.compareAndSet(last, now) || force) {
+      flushAggregations()
+    }


Impact is negligible in practice: force=true is only reached from onApplicationEnd, which fires once at app end with no concurrent onTaskEnd. No extra flushes follow it. Leaving as-is.

yew1eb · 2026-08-18T11:57:45Z

+    <dependency>
+      <groupId>jakarta.servlet</groupId>
+      <artifactId>jakarta.servlet-api</artifactId>
+      <scope>provided</scope>
+    </dependency>


Intentional. A single module pom with ${servlet.source.dir} selecting the source root keeps the build simple; both servlet APIs are provided and only the
one matching the selected source root is ever referenced at compile time. Splitting into per-profile dependencies would duplicate the dependency block
across every Spark profile without real benefit.

yew1eb · 2026-08-18T11:58:25Z

        <module>client-spark/spark-3</module>
        <module>client-spark/spark-3-columnar-common</module>
        <module>client-spark/spark-3-columnar-shuffle</module>
+        <module>client-spark/spark-3-ui</module>


The name follows the existing celeborn-client-spark-3 convention in this repo, which also serves Spark 4. Renaming would diverge from that. Keeping spark-3-ui for consistency.

yew1eb · 2026-08-18T11:59:19Z

+        <span class="collapse-aggregated-celebornProperties collapse-table"
+            onClick="collapseTable('collapse-aggregated-celebornProperties',
+            'aggregated-celebornProperties')">
+          <h4>
+            <span class="collapse-table-arrow arrow-open"></span>
+            <a>Celeborn Properties</a>
+          </h4>
+        </span>
+        <div class="aggregated-celebornProperties collapsible-table">


Following Spark's own UI convention here — Spark uses camelCase collapse identifiers (e.g. collapse-aggregated-allActiveStages). The collapseTable() JS
accepts arbitrary ids as long as the collapse-* prefix and the target class match, which they do. No change.

setupUI() no longer gates on hasData(); the Celeborn tab is now always attached in the History Server, matching the live UI behavior where the tab is attached unconditionally once spark.plugins=CelebornPlugin is set. Removes the now-unused CelebornStatusStore.hasData().

zaynt4606 · 2026-08-19T10:09:48Z

ping @Kalvin2077

Kalvin2077 · 2026-08-31T10:55:40Z

@yew1eb

Could you also add spark-3-ui to the SBT project graph and shaded Spark dependencies? This will ensure the UI classes and History Server SPI are included in release jars.

…encies

yew1eb · 2026-08-31T12:02:11Z

@yew1eb

Could you also add spark-3-ui to the SBT project graph and shaded Spark dependencies? This will ensure the UI classes and History Server SPI are included in release jars.

@Kalvin2077 Thanks for the review! Done.

yew1eb · 2026-09-08T13:53:27Z

@pan3793 @cxzl25 @RexXiong
all review comments have been addressed (thanks @Kalvin2077 for the thorough review) and CI is fully green. Please let me know if there's anything else to improve — otherwise, would you mind taking a final look so we can move this toward merge?
Thanks!

RexXiong · 2026-09-08T14:33:59Z

Two issues to address:

1. Flush accumulated metrics when processing finishes

The 5-second interval in CelebornListener.mayUpdate() only throttles writes when a task event arrives; it does not schedule a flush. For example, task A finishes and flushes the totals, then tasks B and C finish within the next 5 seconds. Their metrics update the in-memory counters, but the writes are skipped. If the SparkSession then remains alive but idle, the store still contains only A's totals, and refreshing the UI does not update them. A normal ApplicationEnd eventually fixes the totals, but a long-lived idle application can keep showing stale values indefinitely.

The same issue affects History Server replay of logs without an ApplicationEnd event: events are processed using wall-clock time, so a replay that finishes within 5 seconds can persist only the first task's totals.

Could we flush at job completion (or periodically) for the live UI, and register an ElementTrackingStore.onFlush callback for the history listener so the final accumulated values are persisted when replay finishes? A regression test with multiple task-end events within the interval, followed by job completion or replay completion without ApplicationEnd, would cover this.

2. Honor the application's plugin opt-in in the History Server

CelebornHistoryServerPlugin is loaded through the AppHistoryServerPlugin SPI, and both createListeners() and setupUI() are unconditional. Since the UI module is included in the shaded client JAR, placing that JAR on the History Server classpath enables collection and adds a Celeborn tab for every replayed application, including applications that never configured spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin. This differs from the live UI and the stated disabled-by-default behavior.

Could we gate metric collection and tab creation on whether the application's recorded spark.plugins list contains org.apache.spark.shuffle.celeborn.CelebornPlugin? The check should use the application configuration from the event log, not the History Server's own configuration. A lightweight listener can inspect the environment event and record whether the extension is enabled for setupUI() to consult. This would make the live and history UIs consistent without introducing another configuration switch. Please also test replay with and without the plugin configured.

…play close; honor plugin opt-in in SHS - Flush per-job via onJobEnd so task metrics landing inside the throttle interval are persisted for long-lived idle applications (live UI). - Register an ElementTrackingStore.onFlush callback so replay persists the final accumulated values even without an ApplicationEnd event. - During replay, gate metric collection on the application's recorded spark.plugins (from the event log, not the History Server's own conf) and persist the opt-in as a KVStore marker; setupUI attaches the Celeborn tab only when the marker is present. - Add CelebornListenerSuite covering both regressions.

yew1eb · 2026-09-08T15:36:36Z

@RexXiong Thanks for the careful review! Both issues are addressed:

Flush on onJobEnd for the live UI, plus an ElementTrackingStore.onFlush callback so replay persists final values even without ApplicationEnd.
Replay now gates collection and the tab on the application's recorded spark.plugins; the opt-in is persisted as a KVStore marker for setupUI (an in-memory flag won't work — FsHistoryProvider reloads plugin instances).

RexXiong · 2026-09-09T01:16:20Z

Thanks for addressing the flush behavior and the History Server opt-in!

One remaining issue: the new CelebornListenerSuite uses JUnit, but sparkClientUi in CelebornBuild.scala does not include the test dependencies. .dependsOn(sparkClient) does not inherit that project's test-scoped dependencies, so SBT CI fails with object junit is not a member of package org.

Could you add commonUnitTestDependencies to sparkClientUi, following the other modules? That should provide both JUnit and the test runner.

…ntUi in SBT build

yew1eb · 2026-09-09T04:01:07Z

@RexXiong Added commonUnitTestDependencies to sparkClientUi. CI is green, PTAL, thanks!

RexXiong

LGTM

cxzl25 · 2026-09-09T10:42:41Z

+    if (!pluginEnabled) {
+      val pluginClass = classOf[CelebornPlugin].getName
+      val optedIn = sparkProps.exists { case (k, v) =>
+        k == "spark.plugins" && v.split(",").exists(_.trim == pluginClass)


Do we need to consider the spark.plugins.defaultList configuration item?

github-actions Bot added module:client module:spark kind:build labels Aug 10, 2026

yew1eb marked this pull request as draft August 10, 2026 11:30

yew1eb marked this pull request as ready for review August 10, 2026 13:38

yew1eb force-pushed the celeborn-ui-mvp branch 2 times, most recently from 4f8bad3 to 68fc2f2 Compare August 10, 2026 13:40

yew1eb marked this pull request as draft August 10, 2026 13:48

yew1eb force-pushed the celeborn-ui-mvp branch 6 times, most recently from 7a9e389 to 47f3676 Compare August 10, 2026 16:27

yew1eb closed this Aug 10, 2026

yew1eb reopened this Aug 10, 2026

[CELEBORN-2415][FOLLOWUP] Add Celeborn Spark UI extension

e053012

yew1eb force-pushed the celeborn-ui-mvp branch from 47f3676 to e053012 Compare August 10, 2026 16:57

yew1eb marked this pull request as ready for review August 10, 2026 17:30

cxzl25 requested a lite review from Copilot August 18, 2026 10:34

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot started reviewing on behalf of cxzl25 August 18, 2026 11:03 View session

yew1eb force-pushed the celeborn-ui-mvp branch from c0cd150 to 18cf1d1 Compare August 18, 2026 12:08

yew1eb closed this Aug 18, 2026

yew1eb reopened this Aug 18, 2026

[CELEBORN-XXXX] Add spark-3-ui to SBT project graph and shaded depend…

191b667

…encies

Kalvin2077 approved these changes Aug 31, 2026

View reviewed changes

Trigger CI

f4bca9d

[CELEBORN-2415][FOLLOWUP] Add commonUnitTestDependencies to sparkClie…

fec19cf

…ntUi in SBT build

RexXiong approved these changes Sep 9, 2026

View reviewed changes

cxzl25 reviewed Sep 9, 2026

View reviewed changes

Uh oh!

Conversation

yew1eb commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Does this PR resolve a correctness bug?

How was this patch tested?

Uh oh!

yew1eb commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

zaynt4606 commented Aug 19, 2026

Uh oh!

Kalvin2077 commented Aug 31, 2026

Uh oh!

yew1eb commented Aug 31, 2026

Uh oh!

yew1eb commented Sep 8, 2026

Uh oh!

RexXiong commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yew1eb commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RexXiong commented Sep 9, 2026

Uh oh!

yew1eb commented Sep 9, 2026

Uh oh!

RexXiong left a comment

Choose a reason for hiding this comment

Uh oh!

cxzl25 Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yew1eb commented Aug 10, 2026 •

edited

Loading

yew1eb Aug 18, 2026 •

edited

Loading

RexXiong commented Sep 8, 2026 •

edited

Loading

yew1eb commented Sep 8, 2026 •

edited

Loading