branch-4.0: [fix](filesystem) Credential-aware hadoop FileSystem cache via doris.fs.cache.key#65756
Open
CalvinKirs wants to merge 1 commit into
Open
branch-4.0: [fix](filesystem) Credential-aware hadoop FileSystem cache via doris.fs.cache.key#65756CalvinKirs wants to merge 1 commit into
CalvinKirs wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
…e via doris.fs.cache.key (apache#65586) Issue Number: close #xxx Related PR: #xxx Problem Summary: Hadoop's `FileSystem.CACHE` keys entries only by `(scheme, authority, UGI)`. Different catalogs/TVFs accessing the same bucket/namenode with different credentials could receive each other's cached `FileSystem` instance (cross-credential contamination), so Doris forced `fs.<schema>.impl.disable.cache=true` everywhere — which makes every access (e.g. every JNI scanner split) create a brand-new `FileSystem` instance (Closing/releasing depends on GC.) This PR makes the cache credential-aware instead of disabling it: **BE/FE shared patch (hadoop-deps):** - Ship a patched copy of `org.apache.hadoop.fs.FileSystem` (verbatim from hadoop-common 3.4.2, hunks marked `DORIS-PATCH`): `Cache.Key` carries one extra dimension read from the reserved conf property `doris.fs.cache.key`. Absent/empty value = byte-for-byte vanilla behavior. - Load-order guarantees: BE loads `be/lib/hadoop_hdfs/hadoop-deps.jar` before `lib/*.jar` (existing); FE now declares `hadoop-deps` as fe-core's first dependency (all transitives excluded) and `start_fe.sh` prepends the jar explicitly, so both JVMs (including JNI scanners and libhdfs) see the patched class. - maven-enforcer pins `hadoop.version=3.4.2`: upgrading hadoop fails the build until the patched file is re-synced. **FE fingerprint injection:** - `StorageProperties` computes a stable 32-hex SHA-256 fingerprint from the concrete class name + `matchedProperties` (the user-supplied storage definition, credentials included) and injects it as `doris.fs.cache.key` through a final `getBackendConfigProperties()` wrapper (subclasses now implement `doGetBackendConfigProperties()`) and through `hadoopStorageConfig`. - Runtime/vended credentials chain into the fingerprint so per-session credentials never share a slot. - Every multi-storage merge site (CatalogProperty backend/hadoop maps, CredentialUtils, Iceberg/Paimon metastore `Configuration` builders, Iceberg table/delete/merge sinks) replaces the last-wins key with an order-independent combined fingerprint via `StorageProperties.setCombinedFsCacheKey`. - The blanket `fs.<schema>.impl.disable.cache=true` defaults are removed; explicit user-provided values are still honored as an escape hatch. FE `DFSFileSystem` keeps its per-instance disable: it exists for manual lifecycle management (RemoteFSPhantomManager closes evicted instances), not credential isolation. **Notes:** - Rolling upgrade: upgrade BE before FE (standard order). Old FE + new BE stays safe because old FE still sends the disable flags; new FE + old BE would re-expose the vanilla cache. - `FileSystem.CACHE` has no eviction, so rotating vended credentials accumulate entries over time — still strictly better than today's new-instance-per-access behavior. Hadoop FileSystem cache is now enabled and keyed by a per-storage credential fingerprint (`doris.fs.cache.key`) instead of being disabled globally; explicit `fs.<schema>.impl.disable.cache` settings are still honored. - Test - [x] Unit test: `DorisFileSystemCacheKeyTest` (patched class loaded, vanilla default behavior, distinct keys isolate, equal keys share), `StoragePropertiesFsCacheFingerprintTest` (stability, credential sensitivity, both injection channels, combined fingerprint semantics), updated GCS/S3/COS/OBS/OSS property tests - [ ] Regression test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. - Behavior changed: - [ ] No. - [x] Yes. Hadoop FileSystem caching is enabled again with credential-aware keys; blanket cache disabling removed. - Does this need documentation? - [ ] No. - [x] Yes. `doris.fs.cache.key` is a reserved property; upgrade order note (BE before FE). - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
CalvinKirs
force-pushed
the
branch-4.0-fs-cache
branch
from
July 17, 2026 07:39
c69d4c8 to
ad1330f
Compare
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Member
Author
|
run feut |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#65586