[GLUTEN-12564][CORE] Exclude DefaultAllocationManagerFactory from gluten-package shade bundle

[wangyum]

What changes are proposed in this pull request?

Exclude DefaultAllocationManagerFactory.class (in all three known Arrow locations — CORE path, unsafe/ subpath, netty/ subpath) from the gluten-package.jar shade bundle.

The class is bundled today because the shade plugin's relocation excludes org.apache.arrow.memory.** from relocation but still includes the class files in the bundle. The class is redundant — the same class is always available on the runtime classpath via the upstream arrow-memory-unsafe / arrow-memory-netty jar declared as a gluten-arrow dependency.

Starting with Arrow 16.0.0, CheckAllocator.check() inspects the URL of DefaultAllocationManagerFactory.class and throws IllegalStateException: Unknown allocation manager type to infer when the URL does not contain memory-core, memory-unsafe, or memory-netty. When the class is loaded from inside gluten-package.jar the URL matches none of these patterns (especially for Arrow <= 15.x where the class lives at the CORE path org/apache/arrow/memory/DefaultAllocationManagerFactory.class).

This is currently silent on upstream CI because:

• Spark 3.3/3.4/3.5 community uses Arrow 12.0.1 whose CheckAllocator predates the URL check.
• Spark 4.0/4.1 uses Arrow 18.1.0; the spark-4.0 profile overrides arrow.version to 18.1.0 so gluten-package bundles the unsafe/-subpath class whose URL contains /org/apache/arrow/memory/unsafe/ and matches.

But the bug manifests whenever a Spark distribution ships Arrow >= 16.0.0 while Gluten is built with arrow.version <= 15.x — e.g. a downstream Spark 3.5 fork that upgrades Arrow from 12.0.1 to 18.3.0 while Gluten's spark-3.5 profile still uses arrow.version=15.0.0. Removing the duplicate class lets the classloader find the upstream Arrow jar, whose URL contains a recognized substring.

The class is functionally identical (same FACTORY field providing an AllocationManager.Factory). The only behavioral change is that Gluten stops forcing the unsafe allocator and lets Arrow's own default selection apply (which is netty per DefaultAllocationManagerOption.java:80).

How was this patch tested?

• Verified the bundled jar before/after the change confirms the class is no longer present in gluten-package.jar.
• Verified the upstream arrow-memory-netty-*.jar / arrow-memory-unsafe-*.jar always provides a DefaultAllocationManagerFactory.class whose URL matches CheckAllocator's pattern (for Arrow >= 16.x). For Arrow <= 15.x the CheckAllocator does not perform URL inspection, so the fix is a no-op there.
• Existing gluten-ut suites continue to pass (no source-code change).

Closes #12564.

---

Was this patch authored or co-authored using generative AI tooling?

No

[Copilot]

Pull request overview

This PR updates the gluten-package shaded bundle configuration to exclude DefaultAllocationManagerFactory.class from being packaged into gluten-package.jar, preventing Arrow 16.0.0+ CheckAllocator URL-based inference failures when Spark ships a newer Arrow version than Gluten’s build-time Arrow.

Changes:

• Exclude DefaultAllocationManagerFactory.class from the shade bundle for the CORE, unsafe/, and netty/ Arrow class paths.
• Add inline documentation explaining the Arrow 16.0.0+ CheckAllocator behavior and why the exclusion is necessary.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.