[FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) #25899

ryanvanhuuksloot · 2025-01-06T03:57:01Z

JIRA Ticket: https://issues.apache.org/jira/browse/FLINK-37008

What is the purpose of the change

It would be useful for the UI to show if a checkpoint is full or incremental. I'm curious how others would like to expose this in the UI / API but I wanted to do a first pass to get the conversation rolling.
This PR is meant to be throwaway when we decided on a path forward.
There are currently no tests and likely fails CI

UPDATED IMAGE

Brief change log

Adds "full checkpoint flag" to checkpoints to be displayed in the UI / API

Verifying this change

N/A for this draft PR

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (kind of)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (not documented)

flinkbot · 2025-01-06T04:03:59Z

CI report:

917fe0f Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

...web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html

Zakelly

Thanks for the PR! overall LGTM

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SnapshotType.java

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SavepointType.java

ryanvanhuuksloot · 2025-01-13T22:27:34Z

I added one test. I looked at other flags on Checkpoints and Savepoints and couldn't tests.

I also couldn't find ui tests but maybe I'm blind.

Let me know if I missed anything!

ryanvanhuuksloot · 2025-01-13T22:40:29Z

@gaborgsomogyi If you have a chance to review given you post in the mailing list

ryanvanhuuksloot · 2025-01-13T22:54:44Z

@flinkbot run azure

Zakelly

Thanks for the update! Please find my comments below:

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SavepointType.java

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java

Zakelly · 2025-01-14T12:33:19Z

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java

+    private static final CheckpointProperties FULL_CHECKPOINT_NEVER_RETAINED =
+            new CheckpointProperties(
+                    false,
+                    CheckpointType.FULL_CHECKPOINT,
+                    true,
+                    true, // Delete on success
+                    true, // Delete on cancellation
+                    true, // Delete on failure
+                    true, // Delete on suspension
+                    false);
+
+    private static final CheckpointProperties FULL_CHECKPOINT_RETAINED_ON_FAILURE =
+            new CheckpointProperties(
+                    false,
+                    CheckpointType.FULL_CHECKPOINT,
+                    true,
+                    true, // Delete on success
+                    true, // Delete on cancellation
+                    false, // Retain on failure
+                    true, // Delete on suspension
+                    false);
+
+    private static final CheckpointProperties FULL_CHECKPOINT_RETAINED_ON_CANCELLATION =
+            new CheckpointProperties(
+                    false,
+                    CheckpointType.FULL_CHECKPOINT,
+                    true,
+                    true, // Delete on success
+                    false, // Retain on cancellation
+                    false, // Retain on failure
+                    false, // Retain on suspension
+                    false);
+


Why those are introduced?

They are only used for tests
We don't really have an test suite for testing this kind of thing

I'm open to ideas for where to add tests
I've thought about
SnapshotUtilsTest
and
CheckpointPropertiesTest

Ah, I see.

I read through the CheckpointCoordinator and it seems the JM will always propagate incremental checkpoints periodically (not the ones triggered by Rest API). But actually the TM will do incremental or full ones by configuration it reads (create RocksNativeFullSnapshotStrategy or RocksIncrementalSnapshotStrategy when state backend build). Meaning that in UI will always show incremental cps even if we disable that. Am I right? If so, we should make some change in CheckpointCoordinator.

I agree looking at the CheckpointCoordinator.

flink/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java

Lines 355 to 356 in 55de8d6

this.checkpointProperties =

CheckpointProperties.forCheckpoint(chkConfig.getCheckpointRetentionPolicy());

Specifically, the issue is that Checkpoint can be used for both full or incremental.

flink/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java

Lines 29 to 31 in 55de8d6

/** A checkpoint, full or incremental. */

public static final CheckpointType CHECKPOINT =

new CheckpointType("Checkpoint", SharingFilesStrategy.FORWARD_BACKWARD);

We'd have to move the evaluation of execution.checkpointing.incremental or change how we are determining if a Checkpoint is Full 🤔

I thought a possible solution is to evaluate execution.checkpointing.incremental in CheckpointCoordinator and set checkpointProperties properly. WDTY?

ryanvanhuuksloot · 2025-01-14T16:35:14Z

@flinkbot run azure

ryanvanhuuksloot · 2025-01-15T14:38:34Z

@flinkbot run azure

ryanvanhuuksloot · 2025-01-15T14:39:44Z

The flink tests seem to fail on a different flaky test. Restarted to make sure.

ryanvanhuuksloot · 2025-01-15T14:40:31Z

@Zakelly I'd love to get this in as part of Flink 2.0. I know you're working on a bunch but if you had some time 🙏
Happy to commit to fast code changes based on PR comments.

Zakelly · 2025-01-16T07:35:26Z

@Zakelly I'd love to get this in as part of Flink 2.0. I know you're working on a bunch but if you had some time 🙏 Happy to commit to fast code changes based on PR comments.

No worries. I will spare some time and we'll make it.

ryanvanhuuksloot · 2025-01-16T16:55:57Z

@flinkbot run azure

ryanvanhuuksloot · 2025-02-25T21:49:25Z

We realized that it is quite difficult to determine at the correct layer if a checkpoint is full when it isn't explicitly marked as full. This needs a more in-depth dive into the code.

ryanvanhuuksloot added 2 commits January 5, 2025 22:49

First attempt at Full Checkpoint information on the UI

028bfba

Add an example UI Change

1494ec2

flinkbot added component=Runtime/WebFrontend component=Runtime/Checkpointing labels Jan 6, 2025

davidradl reviewed Jan 6, 2025

View reviewed changes

...web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html Outdated Show resolved Hide resolved

[FLINK-37008] Swap to 'incremental'

c362d7a

ryanvanhuuksloot force-pushed the rvh.ui-full-checkpoints branch from ffa5245 to c362d7a Compare January 7, 2025 22:03

Zakelly reviewed Jan 13, 2025

View reviewed changes

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SnapshotType.java Outdated Show resolved Hide resolved

[FLINK-37008] Address comments

ca8b457

ryanvanhuuksloot commented Jan 13, 2025

View reviewed changes

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/SavepointType.java Outdated Show resolved Hide resolved

[FLINK-37008] Swap strategy logic

e132d2b

ryanvanhuuksloot marked this pull request as ready for review January 13, 2025 22:26

ryanvanhuuksloot requested review from Zakelly and davidradl January 13, 2025 22:27

Zakelly reviewed Jan 14, 2025

View reviewed changes

ryanvanhuuksloot added 2 commits January 14, 2025 11:34

[FLINK-37008] PR Comments

5e87656

[FLINK-37008] Remove analytics

6d03515

ryanvanhuuksloot requested a review from Zakelly January 14, 2025 16:55

[FLINK-37008] Fix CI

4ba9503

Not sure what is in here

917fe0f

ryanvanhuuksloot closed this Feb 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) #25899

[FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) #25899

ryanvanhuuksloot commented Jan 6, 2025 •

edited

Loading

flinkbot commented Jan 6, 2025 •

edited

Loading

Zakelly left a comment

ryanvanhuuksloot commented Jan 13, 2025

ryanvanhuuksloot commented Jan 13, 2025

ryanvanhuuksloot commented Jan 13, 2025

Zakelly left a comment

Zakelly Jan 14, 2025

ryanvanhuuksloot Jan 14, 2025

Zakelly Jan 16, 2025

ryanvanhuuksloot Jan 16, 2025

Zakelly Jan 17, 2025

ryanvanhuuksloot commented Jan 14, 2025

ryanvanhuuksloot commented Jan 15, 2025

ryanvanhuuksloot commented Jan 15, 2025

ryanvanhuuksloot commented Jan 15, 2025

Zakelly commented Jan 16, 2025

ryanvanhuuksloot commented Jan 16, 2025

ryanvanhuuksloot commented Feb 25, 2025

	this.checkpointProperties =
	CheckpointProperties.forCheckpoint(chkConfig.getCheckpointRetentionPolicy());

	/** A checkpoint, full or incremental. */
	public static final CheckpointType CHECKPOINT =
	new CheckpointType("Checkpoint", SharingFilesStrategy.FORWARD_BACKWARD);

[FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) #25899

[FLINK-37008] [runtime-web] Flink UI should show the type of checkpoint (full vs incremental) #25899

Conversation

ryanvanhuuksloot commented Jan 6, 2025 • edited Loading

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Jan 6, 2025 • edited Loading

CI report:

Zakelly left a comment

Choose a reason for hiding this comment

ryanvanhuuksloot commented Jan 13, 2025

ryanvanhuuksloot commented Jan 13, 2025

ryanvanhuuksloot commented Jan 13, 2025

Zakelly left a comment

Choose a reason for hiding this comment

Zakelly Jan 14, 2025

Choose a reason for hiding this comment

ryanvanhuuksloot Jan 14, 2025

Choose a reason for hiding this comment

Zakelly Jan 16, 2025

Choose a reason for hiding this comment

ryanvanhuuksloot Jan 16, 2025

Choose a reason for hiding this comment

Zakelly Jan 17, 2025

Choose a reason for hiding this comment

ryanvanhuuksloot commented Jan 14, 2025

ryanvanhuuksloot commented Jan 15, 2025

ryanvanhuuksloot commented Jan 15, 2025

ryanvanhuuksloot commented Jan 15, 2025

Zakelly commented Jan 16, 2025

ryanvanhuuksloot commented Jan 16, 2025

ryanvanhuuksloot commented Feb 25, 2025

ryanvanhuuksloot commented Jan 6, 2025 •

edited

Loading

flinkbot commented Jan 6, 2025 •

edited

Loading