Skip to content

HDDS-15930. Avoid zero-length chunk in TestContainerOperations#10833

Open
henrybear327 wants to merge 1 commit into
apache:masterfrom
henrybear327:HDDS-15930
Open

HDDS-15930. Avoid zero-length chunk in TestContainerOperations#10833
henrybear327 wants to merge 1 commit into
apache:masterfrom
henrybear327:HDDS-15930

Conversation

@henrybear327

@henrybear327 henrybear327 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR fixes the flake, which is the TestContainerOperations#testContainerStateMachineIdempotency, by ensuring that its randomly generated chunk is non-empty.

Root cause investigation

Taken from the CI run log.

The failing CI run showed that the test generated a zero-byte chunk:

2026-07-20 21:42:37,572 [ForkJoinPool-1-worker-1] DEBUG scm.XceiverClientGrpc (XceiverClientGrpc.java:sendCommandWithRetry(495)) - Executing command cmdType: WriteChunk traceID: "" containerID: 67 datanodeUuid: "8a3fc17b-fb86-4d79-9be3-3d51133d4f50" writeChunk { blockID { containerID: 67 localID: 116954481136238947 blockCommitSequenceId: 0 } chunkData { chunkName: "116954481136238947.data.0" offset: 0 len: 0 } data: "<redacted>" } version: 3 on datanode 8a3fc17b-fb86-4d79-9be3-3d51133d4f50(localhost/127.0.0.1)

The following PutBlock also described an empty block:

2026-07-20 21:42:37,578 [ForkJoinPool-1-worker-1] DEBUG scm.XceiverClientGrpc (XceiverClientGrpc.java:sendCommandWithRetry(495)) - Executing command cmdType: PutBlock traceID: "" containerID: 67 datanodeUuid: "8a3fc17b-fb86-4d79-9be3-3d51133d4f50" putBlock { blockData { blockID { containerID: 67 localID: 116954481136238947 blockCommitSequenceId: 0 } chunks { chunkName: "116954481136238947.data.0" offset: 0 len: 0 } size: 0 } } version: 3 on datanode 8a3fc17b-fb86-4d79-9be3-3d51133d4f50(localhost/127.0.0.1)

This led to the following failure sequence:

2026-07-20 21:42:37,580 [8a3fc17b-fb86-4d79-9be3-3d51133d4f50-ChunkReader-2] WARN  keyvalue.KeyValueHandler (ContainerUtils.java:logAndReturnError(99)) - Operation: PutBlock , Trace ID:  , Message: Chunk file not found: /home/runner/work/ozone/ozone/hadoop-ozone/integration-test/target/test-dir/MiniOzoneClusterImpl-3f677bbb-a268-48d5-a8e5-e030d249b108/ozone-metadata/datanode-5/data-0/hdds/3f677bbb-a268-48d5-a8e5-e030d249b108/current/containerDir0/67/chunks/116954481136238947.block , Result: UNABLE_TO_FIND_CHUNK , StorageContainerException Occurred.

then

Marked container UNHEALTHY from OPEN

then

Operation: CloseContainer
Message: Cannot close container #67 while in UNHEALTHY state.
Result: CONTAINER_UNHEALTHY

By tracing the code, we can confirm why the chunk file was missing, because FilePerBlockStrategy intentionally skips empty chunk writes (this log was taken from a local reproduction run since this DEBUG log isn't showing up in the CI run log)

2026-07-22 01:08:45,148 [e5111eb4-3873-4f6a-868f-2f14e06a9e36-ChunkReader-6] DEBUG impl.FilePerBlockStrategy (FilePerBlockStrategy.java:writeChunk(134)) - Skip writing empty chunk ChunkInfo{chunkName='116960482106540033.data.0, offset=0, len=0} in stage COMBINED

Therefore, when the test randomly selected zero, WriteChunk created no file, PutBlock failed to find it, and the datanode marked the container unhealthy. The final CloseContainer exception was a downstream symptom.

Proposed fix

The test previously generated a length in the range [0, 1024), just make it non-empty [1, 1024), would work.

As the test is intended to verify container state-machine idempotency, not empty-chunk behavior, so excluding zero keeps its randomized input within the valid scenario being tested.

What is the link to the Apache JIRA

[Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)

If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the Jira guideline.

(Please replace this section with the link to the Apache JIRA)](https://issues.apache.org/jira/browse/HDDS-15930)

How was this patch tested?

https://github.com/apache/ozone/actions/runs/29876934271

https://github.com/henrybear327/ozone/actions/runs/29906444081

@anuragp010 anuragp010 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Chunk length = 0, same error is reproduced.
[ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.216 s <<< FAILURE! -- in org.apache.ozone.test.NonHATests$ContainerOperations [ERROR] org.apache.ozone.test.NonHATests$ContainerOperations.testContainerStateMachineIdempotency -- Time elapsed: 0.048 s <<< ERROR! org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Cannot close container #70 while in UNHEALTHY state.

@henrybear327

Copy link
Copy Markdown
Contributor Author

@chungen0126 I have attempted to enable the flaky-test-check workflow https://github.com/henrybear327/ozone/actions/runs/29906444081. Not sure if this is what you would like to get run :)

Thanks

@chungen0126 chungen0126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM. I'll merge after ci pass

@sarvekshayr sarvekshayr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@henrybear327 Looks like the flaky-test-check(https://github.com/henrybear327/ozone/actions/runs/29906444081) did not run the test at all.

Please trigger another one with the right configs before we merge this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants