Search before asking
Paimon version
master @ 9e306a4
Compute Engine
Engine-agnostic (S3 filesystem plugin); surfaced by Hive clone.
Minimal reproduce step
- Create a HiveCatalog with warehouse
s3://bucket/wh. HiveCatalog.createHiveCatalog builds one FileIO and checkOrMkdirs(warehouse) caches the s3 FileSystem.
- Clone a Hive table whose HMS location is
s3a://bucket/.... CloneFilesFunction and HiveCloneUtils.listFiles read it through the same hiveCatalog.fileIO().
- Fails with
IllegalArgumentException: Wrong FS s3a://bucket/... -expected s3://bucket.
Cause: HadoopCompliantFileIO.getFileSystem() (paimon-s3-impl) keys fsMap by authority only, while each S3AFileSystem is bound to its scheme. Core HadoopFileIO already keys by Pair.of(scheme, authority) (#2504).
What doesn't meet your expectations?
One S3 FileIO should serve both schemes, since the S3 plugin also serves s3a:// (#9814). Actual: the first cached scheme wins.
Anything else?
- Also reachable via
data-file.external-paths with another scheme, only when resolving-file-io.enabled=false; ResolvingFileIO already keys by scheme and authority.
- Cross-scheme rename (e.g. Hive migrate, s3a to s3) still fails; out of scope.
Are you willing to submit a PR?
Search before asking
Paimon version
master @ 9e306a4
Compute Engine
Engine-agnostic (S3 filesystem plugin); surfaced by Hive clone.
Minimal reproduce step
s3://bucket/wh.HiveCatalog.createHiveCatalogbuilds one FileIO andcheckOrMkdirs(warehouse)caches thes3FileSystem.s3a://bucket/....CloneFilesFunctionandHiveCloneUtils.listFilesread it through the samehiveCatalog.fileIO().IllegalArgumentException: Wrong FS s3a://bucket/... -expected s3://bucket.Cause:
HadoopCompliantFileIO.getFileSystem()(paimon-s3-impl) keysfsMapby authority only, while eachS3AFileSystemis bound to its scheme. CoreHadoopFileIOalready keys byPair.of(scheme, authority)(#2504).What doesn't meet your expectations?
One S3 FileIO should serve both schemes, since the S3 plugin also serves
s3a://(#9814). Actual: the first cached scheme wins.Anything else?
data-file.external-pathswith another scheme, only whenresolving-file-io.enabled=false;ResolvingFileIOalready keys by scheme and authority.Are you willing to submit a PR?