HDDS-15930. Avoid zero-length chunk in TestContainerOperations#10833
HDDS-15930. Avoid zero-length chunk in TestContainerOperations#10833henrybear327 wants to merge 1 commit into
Conversation
anuragp010
left a comment
There was a problem hiding this comment.
With Chunk length = 0, same error is reproduced.
[ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.216 s <<< FAILURE! -- in org.apache.ozone.test.NonHATests$ContainerOperations [ERROR] org.apache.ozone.test.NonHATests$ContainerOperations.testContainerStateMachineIdempotency -- Time elapsed: 0.048 s <<< ERROR! org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Cannot close container #70 while in UNHEALTHY state.
|
@chungen0126 I have attempted to enable the flaky-test-check workflow https://github.com/henrybear327/ozone/actions/runs/29906444081. Not sure if this is what you would like to get run :) Thanks |
sarvekshayr
left a comment
There was a problem hiding this comment.
@henrybear327 Looks like the flaky-test-check(https://github.com/henrybear327/ozone/actions/runs/29906444081) did not run the test at all.
Please trigger another one with the right configs before we merge this.
What changes were proposed in this pull request?
This PR fixes the flake, which is the
TestContainerOperations#testContainerStateMachineIdempotency, by ensuring that its randomly generated chunk is non-empty.Root cause investigation
Taken from the CI run log.
The failing CI run showed that the test generated a zero-byte chunk:
The following
PutBlockalso described an empty block:This led to the following failure sequence:
then
then
By tracing the code, we can confirm why the chunk file was missing, because
FilePerBlockStrategyintentionally skips empty chunk writes (this log was taken from a local reproduction run since this DEBUG log isn't showing up in the CI run log)Therefore, when the test randomly selected zero,
WriteChunkcreated no file,PutBlockfailed to find it, and the datanode marked the container unhealthy. The finalCloseContainerexception was a downstream symptom.Proposed fix
The test previously generated a length in the range
[0, 1024), just make it non-empty[1, 1024), would work.As the test is intended to verify container state-machine idempotency, not empty-chunk behavior, so excluding zero keeps its randomized input within the valid scenario being tested.
What is the link to the Apache JIRA
[Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the Jira guideline.
(Please replace this section with the link to the Apache JIRA)](https://issues.apache.org/jira/browse/HDDS-15930)
How was this patch tested?
https://github.com/apache/ozone/actions/runs/29876934271
https://github.com/henrybear327/ozone/actions/runs/29906444081