Skip to content

[Bug] S3 FileIO fails with Wrong FS when one FileIO reads both s3:// and s3a:// paths of the same bucket #10181

Description

@thswlsqls

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version
master @ 9e306a4

Compute Engine
Engine-agnostic (S3 filesystem plugin); surfaced by Hive clone.

Minimal reproduce step

  1. Create a HiveCatalog with warehouse s3://bucket/wh. HiveCatalog.createHiveCatalog builds one FileIO and checkOrMkdirs(warehouse) caches the s3 FileSystem.
  2. Clone a Hive table whose HMS location is s3a://bucket/.... CloneFilesFunction and HiveCloneUtils.listFiles read it through the same hiveCatalog.fileIO().
  3. Fails with IllegalArgumentException: Wrong FS s3a://bucket/... -expected s3://bucket.

Cause: HadoopCompliantFileIO.getFileSystem() (paimon-s3-impl) keys fsMap by authority only, while each S3AFileSystem is bound to its scheme. Core HadoopFileIO already keys by Pair.of(scheme, authority) (#2504).

What doesn't meet your expectations?
One S3 FileIO should serve both schemes, since the S3 plugin also serves s3a:// (#9814). Actual: the first cached scheme wins.

Anything else?

  • Also reachable via data-file.external-paths with another scheme, only when resolving-file-io.enabled=false; ResolvingFileIO already keys by scheme and authority.
  • Cross-scheme rename (e.g. Hive migrate, s3a to s3) still fails; out of scope.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions