[DNM][CORE] Replace string-based native conf key lists with declarative NativeConfRegistry

backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala

@@ -533,6 +533,8 @@ object VeloxConfig extends ConfigRegistry {
 
   val COLUMNAR_VELOX_FILE_HANDLE_CACHE_ENABLED =
     buildStaticConf("spark.gluten.sql.columnar.backend.velox.fileHandleCacheEnabled")
+      .passToNative()
+      .passDefault()
       .doc(
         "Disables caching if false. File handle cache should be disabled " +
           "if files are mutable, i.e. file content may change while file path stays the same.")
@@ -586,24 +588,47 @@ object VeloxConfig extends ConfigRegistry {
 
   val CACHE_PREFETCH_MINPCT =
     buildStaticConf("spark.gluten.sql.columnar.backend.velox.cachePrefetchMinPct")
+      .passToNative()
       .doc("Set prefetch cache min pct for velox file scan")
       .intConf
       .createWithDefault(0)
 
   val AWS_SDK_LOG_LEVEL =
-    buildConf("spark.gluten.velox.awsSdkLogLevel")
+    buildStaticConf("spark.gluten.velox.awsSdkLogLevel")
       .internal()
+      .passToNative()
+      .passDefault()
       .doc("Log granularity of AWS C++ SDK in velox.")
       .stringConf
       .createWithDefault("FATAL")
 
   val AWS_S3_RETRY_MODE =
-    buildConf("spark.gluten.velox.fs.s3a.retry.mode")
+    buildStaticConf("spark.gluten.velox.fs.s3a.retry.mode")
       .internal()
+      .passToNative()
+      .passDefault()
       .doc("Retry mode for AWS s3 connection error: legacy, standard and adaptive.")
       .stringConf
       .createWithDefault("legacy")
 
+  val S3_USE_PROXY_FROM_ENV =
+    buildStaticConf("spark.gluten.velox.s3UseProxyFromEnv")
+      .internal()
+      .passToNative()
+      .passDefault()
+      .doc("Whether to use proxy from environment variables for S3 C++ client.")
+      .booleanConf
+      .createWithDefault(false)
+
+  val S3_PAYLOAD_SIGNING_POLICY =
+    buildStaticConf("spark.gluten.velox.s3PayloadSigningPolicy")
+      .internal()
+      .passToNative()
+      .passDefault()
+      .doc("The S3 payload signing policy: Always, RequestDependent and Never.")
+      .stringConf
+      .createWithDefault("Never")
+
   val S3_UPLOAD_PART_ASYNC =
     buildStaticConf("spark.gluten.velox.s3UploadPartAsync")
       .doc("If true, S3 multipart upload parts are uploaded asynchronously.")
@@ -790,7 +815,8 @@ object VeloxConfig extends ConfigRegistry {
       .createWithDefault(50)
 
   val CUDF_ENABLE_TABLE_SCAN =
-    buildStaticConf("spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan")
+    buildConf("spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan")
+      .passToNative()
       .doc("Enable cudf table scan")
       .booleanConf
       .createWithDefault(false)

docs/developers/NativeConfPassing.md

@@ -0,0 +1,191 @@
+---
+layout: page
+title: Passing Configurations from JVM to Native
+nav_order: 19
+parent: Developer Overview
+---
+
+# Passing Configurations from JVM to Native
+
+This document describes the design of Gluten's JVM-to-native configuration-passing mechanism,
+built around `NativeConfRegistry` and the `ConfigBuilder.passToNative()` declaration API.
+
+## Background and motivation
+
+Native code (Velox / ClickHouse backends) consumes Spark, Hadoop and Gluten configurations.
+Historically, the JVM side decided which keys to deliver via hard-coded string lists inside
+`GlutenConfig.getNativeSessionConf` / `getNativeBackendConf`:
+
+- A `nativeKeys` set of 40+ raw string keys, including backend-specific keys living in common
+  code (e.g. Velox-only S3 keys in gluten-substrait).
+- Two "configs having default values" `Seq`s duplicating each conf's key and default.
+- Ad-hoc per-key special cases (byte-unit conversion, upper-casing, etc.) inline in the
+  selection methods.
+
+This was hard to maintain: adding a native conf required editing central lists far away from
+the conf's definition, defaults were duplicated and could drift, and backend-specific keys
+leaked into common code.
+
+## Core model
+
+### Two delivery channels, two scopes
+
+There are exactly two points where the JVM delivers confs to native, matching the two
+lifecycle stages of a native backend:
+
+| `NativeScope` | Delivery point | JVM entry | Native receiver |
+|---|---|---|---|
+| `BACKEND` | Once, during native backend initialization | `GlutenConfig.getNativeBackendConf` | e.g. Velox `VeloxBackend::init` (`backendConf_`), CH `BackendInitializerUtil` |
+| `RUNTIME` | Each time a native runtime instance is created (per task pipeline / native memory manager) | `GlutenConfig.getNativeSessionConf` | e.g. Velox `VeloxRuntime` (`confMap_` / `veloxCfg_`) |
+
+`NativeScope.ALL` means both.
+
+### Staticness determines the scope
+
+Gluten follows Spark's static/non-static SQL conf semantics on the JVM side, and derives the
+native scope from it:
+
+- A **static** conf (`buildStaticConf`) cannot be modified after startup. Passing it once at
+  backend initialization is therefore lossless: the snapshot taken at init time is the value,
+  forever. Hence **static confs map to `BACKEND`**.
+- A **non-static** conf (`buildConf`) may be changed per session/query. It must be re-delivered
+  each time a native runtime is created so native always observes the current session value.
+  Hence **dynamic confs map to `RUNTIME`**.
+
+Note that this describes the *delivery* semantics, not usability: a `RUNTIME` conf can still be
+set at startup (spark-defaults / `SparkConf`) and the session inherits that value; the point is
+that its native delivery happens at runtime creation, picking up whatever the session currently
+holds.
+
+Conversely, a conf whose declared staticness disagrees with where native actually consumes it
+indicates the declaration is wrong and should be fixed:
+
+1. Declared static, but native reads it per runtime → it should be non-static.
+2. Declared non-static, but native reads it only at backend init and later changes can never
+   take effect → it should be static.
+3. Consumed at both backend init and per runtime → allowed, but must be dual-registered and its
+   split-brain semantics documented (see "Dual-scope confs" below).
+
+## Declaration API
+
+### Gluten confs: builder markers
+
+For configurations defined through Gluten's `ConfigBuilder`, chain the markers at the conf
+definition:
+
+```scala
+val COLUMNAR_MAX_BATCH_SIZE =
+  buildConf("spark.gluten.sql.columnar.maxBatchSize")   // dynamic -> RUNTIME
+    .passToNative()
+    .intConf
+    .createWithDefault(4096)
+
+val COLUMNAR_VELOX_FILE_HANDLE_CACHE_ENABLED =
+  buildStaticConf("spark.gluten.sql.columnar.backend.velox.fileHandleCacheEnabled")
+    .passToNative()      // static -> BACKEND
+    .passDefault()       // native relies on the key being always present
+    .booleanConf
+    .createWithDefault(false)
+```
+
+- `passToNative()`: registers the conf to `NativeConfRegistry` on entry creation. The scope is
+  derived from staticness as described above; there is no scope argument.
+- `passDefault()`: additionally passes the conf's own default value (in parsed form, e.g. a
+  "64MB" bytes conf is passed as "67108864") even when the user did not set the conf. Use it
+  when native relies on the key being always present. It requires `passToNative()` and a
+  defined default value; both constraints are checked eagerly at entry creation.
+
+### Non-Gluten keys: explicit registration
+
+Spark SQL / Spark core / Hadoop keys that have no Gluten `ConfigEntry` are registered
+explicitly, from the conf object of the module that needs them:
+
+```scala
+NativeConfRegistry.register(SQLConf.ANSI_ENABLED.key, RUNTIME, Some(defaultString))
+NativeConfRegistry.register("spark.sql.orc.compression.codec", BACKEND, Some("snappy"))
+NativeConfRegistry.register(
+  SPARK_SHUFFLE_FILE_BUFFER,
+  RUNTIME,
+  transform = Some(v => (JavaUtils.byteStringAs(v, ByteUnit.KiB) * 1024).toString))
+```
+
+`register(key, scope, defaultToPass, transform)`:
+
+- `defaultToPass`: when defined, the value is delivered even if the user did not set the key
+  (same role as `passDefault()` for builder confs).
+- `transform`: normalization applied to user-set values before delivery, e.g. size-string to
+  bytes, upper-casing. Not applied to `defaultToPass` (which is already in final form).
+
+Runtime and backend scopes are tracked separately, so the same key may carry different
+semantics per scope (e.g. plain filter in one scope, always-passed with default in the other).
+Duplicate registration within one scope fails fast.
+
+### Modularity
+
+Registrations live in the conf object of the owning module (`GlutenCoreConfig`,
+`GlutenConfig`, `VeloxConfig`, ...) and take effect when that object is loaded. A backend's or
+connector's registrations therefore never leak into another deployment: Velox-only keys are
+declared in backends-velox and simply do not exist when running the ClickHouse backend.
+
+## Selection at delivery time
+
+- `getNativeSessionConf` = `NativeConfRegistry.selectRuntimeConf(sessionConf)` + prefix rules
+  (`spark.gluten.sql.columnar.backend.<backend>` for non-static keys,
+  `spark.gluten.<backend>`) + UGI tokens.
+- `getNativeBackendConf` = `NativeConfRegistry.selectBackendConf(conf)` + prefix rules
+  (`spark.gluten.sql.columnar.backend.<backend>`, `spark.hadoop.fs.s3a.` / `fs.azure.` /
+  `fs.gs.`, `spark.gluten.<backend>`).
+
+Prefix rules are pattern-based and intentionally kept: they cover open-ended key families
+(e.g. arbitrary `fs.s3a` client options) that cannot be enumerated. The registry covers the
+keys native depends on individually — especially those needing defaults or transforms.
+
+## Dual-scope confs (BACKEND + RUNTIME)
+
+A small set of dynamic confs is also consumed during native backend initialization. They keep
+the builder-derived `RUNTIME` registration and add an explicit `BACKEND` registration. The
+backend-init delivery is a one-time snapshot of the startup value; later session changes only
+affect the runtime channel. Each conf's split semantics:
+
+| Conf | Backend-init consumption (frozen at startup) | Runtime consumption (follows session) |
+|---|---|---|
+| `spark.gluten.sql.debug` | keeps user glog levels | per-task input/plan debug dumps |
+| `spark.gluten.sql.columnar.cudf` | one-time GPU environment initialization | per-query CPU/GPU offload decision. Enabling in-session without startup enablement will not initialize the GPU |
+| `spark.gluten.memory.task.offHeap.size.in.bytes` | CH external sort/aggregation thresholds | Velox per-task spill memory limit |
+| `spark.sql.legacy.statisticalAggregate`, `spark.sql.decimalOperations.allowPrecisionLoss`, `spark.sql.legacy.timeParserPolicy` | expression/aggregate behavior fixed into reused backend structures | per-query expression evaluation |
+| `spark.sql.session.timeZone`, `spark.redaction.regex` | registered `ALL` | registered `ALL` |
+| `spark.hadoop.fs.s3a.*` connection confs (ssl, path-style, retry attempts, connection maximum, ...) | reused HiveConnector construction (defaults passed when unset) | per-query file system access |
+
+Two confs are BACKEND-only on the native side but still must stay non-static on the JVM side,
+solely because `GlutenAutoAdjustStageResourceProfile` mutates them on `SQLConf` at runtime
+when a new resource profile is applied:
+
+- `spark.gluten.memory.offHeap.size.in.bytes`
+- `spark.gluten.numTaskSlotsPerExecutor` (native warns and falls back to 1 when the key is
+  missing, hence its default is always passed)
+
+If the resource-profile adjustment ever moves to a dedicated passing mechanism, these two can
+be tightened to static confs.
+
+Backend note: the ClickHouse backend calls `getNativeBackendConf` both at `initNative` and per
+kernel pipeline, so for CH the BACKEND scope effectively refreshes per query as well; the
+Velox backend consumes BACKEND scope strictly once at init. The scope contract above is
+defined by the stricter (Velox) behavior.
+
+## What was removed
+
+- `nativeKeys` hard-coded string set (including Velox keys in common code, now declared in
+  `VeloxConfig`).
+- The two default-value `Seq`s and all per-key special cases in `getNativeSessionConf` /
+  `getNativeBackendConf`.
+- `BackendSettingsApi.extraNativeSessionConfKeys` / `extraNativeBackendConfKeys`.
+- `GlutenConfigUtil.mapByteConfValue` (superseded by per-entry `transform`).
+
+The selected key/value results of both methods are intended to be byte-for-byte identical to
+the previous implementation.
+
+## Testing
+
+`org.apache.gluten.config.NativeConfRegistrySuite` covers scope derivation from staticness,
+`passDefault` constraints, per-scope mixed semantics, duplicate-registration rejection,
+`defaultToPass` and `transform` behavior.