Search before asking
Paimon version
master (645ce6f)
Compute Engine
Flink
Minimal reproduce step
AppendTableITCase.testCompactionInStreamingMode, and its sibling testCompactionInStreamingModeWithMaxWatermark, run a streaming INSERT from a datagen source at rows-per-second = 1 into an append table configured with compaction.min.file-num = 4, continuous.discovery-interval = 1 s, and a 500 ms checkpoint interval. They then poll once a second, for up to 60 s, for a snapshot whose commitKind is COMPACT.
In CI the poll sometimes reaches the 60 s limit and the test fails:
java.lang.RuntimeException: Time up for streaming execute, don't get expected result.
One example: https://github.com/apache/paimon/actions/runs/36149147465/job/108117843839
What doesn't meet your expectations?
The test should not time out. Nothing is broken in the append-compaction pipeline; the failure is a timing race inside the test.
On a healthy run the first COMPACT snapshot arrives in under 20 s: the streaming job starts, four small files accumulate at about one per second, the coordinator picks them up on its next discovery cycle, compaction runs, and the compacted snapshot commits. The 60 s budget is usually comfortable. Under CI load, streaming job deployment and checkpoint/commit progress can stall, and the first COMPACT snapshot occasionally shows up after 60 s.
The same test was reported in #1634 and closed without a fix.
Anything else?
#1634 suggested lowering compaction.min.file-num as a workaround. That only shortens the file-accumulation phase (roughly 4 s at one file per second), which is not where the time goes when the test times out. The stall is in job deployment and checkpoint progress, so giving the wait a larger timeout is the direct fix.
Are you willing to submit a PR?
Search before asking
Paimon version
master (645ce6f)
Compute Engine
Flink
Minimal reproduce step
AppendTableITCase.testCompactionInStreamingMode, and its siblingtestCompactionInStreamingModeWithMaxWatermark, run a streamingINSERTfrom adatagensource atrows-per-second = 1into an append table configured withcompaction.min.file-num = 4,continuous.discovery-interval = 1 s, and a 500 ms checkpoint interval. They then poll once a second, for up to 60 s, for a snapshot whosecommitKindisCOMPACT.In CI the poll sometimes reaches the 60 s limit and the test fails:
One example: https://github.com/apache/paimon/actions/runs/36149147465/job/108117843839
What doesn't meet your expectations?
The test should not time out. Nothing is broken in the append-compaction pipeline; the failure is a timing race inside the test.
On a healthy run the first
COMPACTsnapshot arrives in under 20 s: the streaming job starts, four small files accumulate at about one per second, the coordinator picks them up on its next discovery cycle, compaction runs, and the compacted snapshot commits. The 60 s budget is usually comfortable. Under CI load, streaming job deployment and checkpoint/commit progress can stall, and the firstCOMPACTsnapshot occasionally shows up after 60 s.The same test was reported in #1634 and closed without a fix.
Anything else?
#1634 suggested lowering
compaction.min.file-numas a workaround. That only shortens the file-accumulation phase (roughly 4 s at one file per second), which is not where the time goes when the test times out. The stall is in job deployment and checkpoint progress, so giving the wait a larger timeout is the direct fix.Are you willing to submit a PR?