HADOOP-19658. ABFS:Create and rename idempotency for FNS Blob #7914
+508
−135
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anmolanmol1234
changed the title
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| AbfsRestOperation statusOp = null; | ||
| try { | ||
| // Check if the file already exists by calling GetPathStatus | ||
| statusOp = getPathStatus(path, false, tracingContext, null); |
There was a problem hiding this comment.
In case of override true, flow might come here with already a Head call done on path.
Can we avoid this head call in that case?
| assumeHnsDisabled(); | ||
| // Create a spy of AzureBlobFileSystemStore | ||
| AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore()); | ||
| assumeBlobServiceType(); |
There was a problem hiding this comment.
We can move all assume before any other statement
| // Create a spy of AzureBlobFileSystem | ||
| try (AzureBlobFileSystem fs = Mockito.spy( | ||
| (AzureBlobFileSystem) FileSystem.newInstance(getRawConfiguration()))) { | ||
| assumeHnsDisabled(); |
There was a problem hiding this comment.
Same here, move all assume to first few lines
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
anujmodi2021
approved these changes
Sep 1, 2025
| @@ -427,6 +427,8 @@ public static String containerProperty(String property, String fsName, String ac | |||
| public static final String FS_AZURE_BLOB_DIR_DELETE_MAX_THREAD = "fs.azure.blob.dir.delete.max.thread"; | |||
| /**Flag to enable/disable sending client transactional ID during create/rename operations: {@value}*/ | |||
| public static final String FS_AZURE_ENABLE_CLIENT_TRANSACTION_ID = "fs.azure.enable.client.transaction.id"; | |||
| /**Flag to enable/disable create idempotency during create operation: {@value}*/ | |||
| public static final String FS_AZURE_ENABLE_CREATE_IDEMPOTENCY = "fs.azure.enable.create.idempotency"; | |||
There was a problem hiding this comment.
This is only for Blob Idempotency, may be we can keep config name accordingly
| @@ -239,5 +239,7 @@ public final class FileSystemConfigurations { | |||
|
|
|||
| public static final boolean DEFAULT_FS_AZURE_ENABLE_CLIENT_TRANSACTION_ID = true; | |||
|
|
|||
| public static final boolean DEFAULT_FS_AZURE_ENABLE_ENABLE_CREATE_IDEMPOTENCY = true; | |||
There was a problem hiding this comment.
Typo, ENABLE added twice
| @@ -54,7 +56,15 @@ public static void dumpHeadersToDebugLog(final String origin, | |||
| if (key == null) { | |||
| key = "HTTP Response"; | |||
| } | |||
| String values = StringUtils.join(";", entry.getValue()); | |||
| List<String> valuesList = entry.getValue(); | |||
There was a problem hiding this comment.
Due to null pointer exceptions on enabling AbfsIoUtils logging if value is null.
|
🎊 +1 overall
This message was automatically generated. |
| @@ -1043,6 +1047,12 @@ public String getAzureAtomicRenameDirs() { | |||
| } | |||
|
|
|||
| public boolean isConditionalCreateOverwriteEnabled() { | |||
| // If either the configured FS service type or the ingress service type is BLOB, | |||
| // conditional create-overwrite is not used. | |||
| if (getFsConfiguredServiceType() == AbfsServiceType.BLOB | |||
There was a problem hiding this comment.
why is this change needed?
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
manika137
approved these changes
Sep 2, 2025
============================================================
|
anmolanmol1234
added a commit
to anmolanmol1234/hadoop
that referenced
this pull request
Sep 3, 2025
…#7914) Contributed by Anmol Asrani
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change introduces idempotency for create operation in the ABFS driver when using FNS-Blob. The update strengthens the client’s retry semantics to ensure that transient backend failures (e.g., HTTP 500) or ambiguous outcomes (where the client receives an error, but the operation has already succeeded on the service) are handled safely. This PR also includes test cases to validate the idempotency of rename operations.