[DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry by jackylee-ch · Pull Request #12549 · apache/gluten

jackylee-ch · 2026-07-17T10:16:00Z

What changes were proposed in this pull request?

[DNM] Do Not Merge — for design discussion.

This PR fully replaces the hard-coded, string-based native conf key lists in GlutenConfig.getNativeSessionConf / getNativeBackendConf with a declarative registration mechanism, so each module declares its own native confs on demand instead of common code maintaining giant key lists.

The mechanism

NativeConfRegistry (gluten-core): a central registry of conf keys to pass to native side, keyed by NativeScope:
- RUNTIME: dynamic, passed each time a native runtime instance is created;
- BACKEND: static, passed once during native backend initialization;
- ALL: both.
Runtime and backend scopes are tracked separately, so the same key can carry different semantics per scope. Each entry supports:
- defaultToPass: if defined, the key is always passed, using the default value when unset by user (replaces the old "configs having default values" lists);
- transform: applied to user-set values before passing (replaces the old per-key special cases such as byte-unit conversion for spark.shuffle.file.buffer and upper-casing for spark.sql.legacy.timeParserPolicy).
ConfigBuilder.passToNative(): Gluten config entries declare native passing right at their definitions. No scope argument — the native scope is derived from the conf's staticness: static confs (buildStaticConf) go to BACKEND, dynamic confs (buildConf) go to RUNTIME:

val COLUMNAR_MAX_BATCH_SIZE =
  buildConf("spark.gluten.sql.columnar.maxBatchSize")
    .passToNative()   // dynamic conf -> passed on each native runtime creation
    .intConf
    .createWithDefault(4096)

val DEBUG_CUDF =
  buildStaticConf("spark.gluten.sql.debug.cudf")   // static conf -> passed at backend init
    .passToNative()
    .passDefault()    // native relies on the key: pass the default when unset
    .booleanConf
    .createWithDefault(false)

passDefault() marks that the entry's own default value (parsed form, e.g. a bytesConf default "1KB" is delivered as "1024") is passed even when the conf is not set by user.

Confs whose native consumption scope differs from the derived one (e.g. static velox cachePrefetchMinPct read on runtime creation; dynamic memory-sizing confs consumed at backend init) register explicitly via NativeConfRegistry.register at the end of their conf object body, with comments explaining the mismatch.

NativeConfRegistry.register: for non-Gluten keys (SQLConf / Spark core / Hadoop / S3 / GCS) that have no Gluten ConfigEntry, and for the scope-mismatch cases above. Common registrations live in registerNativeConfs() in GlutenConfig; velox-only keys are registered from VeloxConfig in backends-velox instead of common code. Registrations are naturally modular: a backend's registrations only take effect when its conf object loads, so backend-specific keys never leak across deployments.

What is removed

The nativeKeys set (40+ hard-coded strings, including velox-specific keys living in common code) in GlutenConfig.
The two "configs having default values" Seqs and all per-key special-case code in getNativeSessionConf / getNativeBackendConf. Both methods now do: registry selection + existing prefix rules + UGI tokens (session only).
BackendSettingsApi.extraNativeSessionConfKeys / extraNativeBackendConfKeys (no backend ever overrode them).
GlutenConfigUtil.mapByteConfValue (superseded by entry-level transform).

Behavior compatibility

The selected key/value results of getNativeSessionConf / getNativeBackendConf are intended to be identical to before. Notable equivalence mappings:

Old code	New declaration
`nativeKeys` filter	`.passToNative()` on dynamic confs / `register(key, RUNTIME)`
session default-value `Seq`	`.passToNative().passDefault()` / `register(..., defaultToPass)`
`mapByteConfValue` special cases	`register(..., transform = byte conversion)`
`timeParserPolicy.toUpperCase`	`register(..., transform = upper-case)`
backend default-value `Seq`	`.passToNative().passDefault()` on static confs / `register(..., BACKEND, defaultToPass)`
backend `keys` filter	`.passToNative()` on static confs / `register(key, BACKEND)`
velox keys in common code	declared in `VeloxConfig` (backends-velox)

Prefix-based rules (spark.gluten.sql.columnar.backend.<backend>, spark.hadoop.fs.s3a. etc.) are kept as-is, since they are pattern rules rather than string key registrations.

Staticness alignment with native consumption scope

Static confs are for native backend initialization (immutable afterwards); non-static confs are for native session. Confs whose declared staticness did not match their actual native consumption were fixed:

Static but consumed per native runtime -> made non-static
- spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan: read by native in WholeStageResultIterator / SubstraitToVeloxPlan on every query, not at backend init, so it is session-tunable now.
Non-static but consumed only at native backend init -> made static
- spark.gluten.velox.awsSdkLogLevel, spark.gluten.velox.fs.s3a.retry.mode: only read when building the reused native HiveConnector at backend init; session changes never took effect.
- spark.gluten.memoryOverhead.size.in.bytes: only set on SparkConf at driver start (VeloxListenerApi), consumed by Velox backend init for global memory sizing.
- spark.gluten.velox.s3UseProxyFromEnv, spark.gluten.velox.s3PayloadSigningPolicy: promoted from raw string registrations to static ConfigEntrys in VeloxConfig.
Non-static and consumed by both backend init and session -> allowed, dual-registered and documented in code

Conf	Backend-init consumption	Session consumption	Special note
`spark.gluten.sql.debug`	keep user glog levels	per-task debug dumps	log level frozen at startup; session toggle only affects per-query dumps
`spark.gluten.sql.columnar.cudf`	one-time GPU env init	per-query CPU/GPU offload decision	enabling in session without startup enablement will not initialize GPU
`spark.gluten.memory.task.offHeap.size.in.bytes`	CH external sort/agg settings	Velox per-task spill limit	mutated at runtime by `GlutenAutoAdjustStageResourceProfile`
`spark.gluten.memory.offHeap.size.in.bytes`	CH spill thresholds	(JVM-side only)	must stay non-static for resource-profile adjustment; BACKEND-only native registration
`spark.gluten.numTaskSlotsPerExecutor`	Velox io/spill thread sizing (key must always be present, default passed when unset)	(JVM-side only)	same as above

Design document

docs/developers/NativeConfPassing.md documents the full design: the two delivery channels (BACKEND at native backend init, RUNTIME at native runtime creation), scope derivation from conf staticness, the passToNative()/passDefault() builder API, explicit NativeConfRegistry.register for non-Gluten keys, prefix rules, and the enumerated dual-scope confs with their split semantics.

How was this patch tested?

New/extended NativeConfRegistrySuite covering: scope derivation from conf staticness, parsed-default registration for bytes confs, passDefault constraint checks, per-scope mixed semantics, duplicate registration rejection, defaultToPass and transform semantics.
Existing config suites in gluten-core / gluten-substrait pass.
Compiled gluten-core, gluten-substrait, backends-velox with -Pspark-3.5,backends-velox (checkstyle/scalastyle/spotless clean). backends-clickhouse Java code compiles; the delta33 Scala compile failure on this machine is pre-existing and unrelated.

AI disclosure: this PR was developed with the assistance of Cursor (Fable 5), with human review of all changes.

…ister native confs Currently, conf keys passed to native side are maintained in hard-coded string lists (`nativeKeys` in GlutenConfig, `extraNative*ConfKeys` in backend settings) that are decoupled from where the configs are defined. This introduces an additive registration mechanism: - `ConfigBuilder.passToNative(scope)` marks a Gluten config entry to be passed to native side, registered automatically on entry creation. - `NativeConfScope` (SESSION/BACKEND/BOTH) declares whether the conf is passed on each native runtime creation, once at backend init, or both. - `NativeConfRegistry` collects registrations; raw Spark/Hadoop keys can be registered via `registerRaw`, optionally with a default value that is always passed. - `getNativeSessionConf`/`getNativeBackendConf` additionally include registered confs, without changing existing key lists or prefix rules. This enables incremental migration of the hard-coded lists and allows each module to register its own native confs on demand. Generated-by: Claude (Cursor Agent)

github-actions · 2026-07-17T10:16:30Z