[SPARK-58295][SQL][PYTHON] Add to_base32 and from_base32 functions

[SreeramaYeshwanthGowd]

What changes were proposed in this pull request?

Add a matched pair of built in scalar functions:

• to_base32(binary) encodes bytes as an RFC 4648 Base32 string.
• from_base32(string) decodes an RFC 4648 Base32 string back to binary.

Spark already ships hex/unhex (base16) and base64/unbase64, so this completes the standard base16/base32/base64 encoding ladder.

API surface added:

• SQL: to_base32(binary) and from_base32(string)
• Scala DataFrame: functions.to_base32(col) and functions.from_base32(col)
• PySpark, classic and Spark Connect: pyspark.sql.functions.to_base32 and from_base32

Key design choices:

• Implemented as RuntimeReplaceable expressions backed by a StaticInvoke, mirroring the existing base64/unbase64 functions, so there is no hand written codegen.
• The Base32 codec delegates to org.apache.commons.codec.binary.Base32 (RFC 4648), the same way base64 delegates to java.util.Base64. java.util.Base64 has no Base32 support, and commons-codec is already a dependency of the catalyst module, so this adds no new dependency.
• The SQL names to_base32 and from_base32 match Trino and BigQuery, and follow Spark's established to_*/from_* conversion family (for example to_json/from_json, to_binary). No major engine uses base32/unbase32.
• from_base32 decodes leniently, ignoring characters outside the Base32 alphabet, consistent with the existing unbase64 behavior. A strict variant is out of scope, matching how unbase64 defers strict decoding to to_binary.
• No SparkR binding is added, consistent with all recently added functions (for example to_binary, is_valid_utf8, luhn_check), since SparkR is deprecated as of Spark 4.0 and its function surface is frozen.

Why are the changes needed?

Base32 (RFC 4648) is the standard encoding for TOTP and 2FA shared secrets (RFC 6238), DNS NSEC3 records, and various identifier schemes. Spark has base16 (hex) and base64 but no base32, so users fall back to UDFs. Trino (to_base32/from_base32) and Google BigQuery (TO_BASE32/FROM_BASE32) both ship the pair, so this also improves parity with the engines Spark users migrate from.

Does this PR introduce any user-facing change?

Yes. It adds two new built in SQL functions, to_base32 and from_base32, and the corresponding Scala and PySpark DataFrame API entries. No existing behavior changes.

Example:

spark-sql> SELECT to_base32(encode('foobar', 'utf-8'));
MZXW6YTBOI======
spark-sql> SELECT cast(from_base32('MZXW6YTBOI======') as string);
foobar

How was this patch tested?

Added catalyst unit tests in StringExpressionsSuite covering the RFC 4648 test vectors, round trip encode and decode, empty input, and null propagation, and a DataFrame API test in StringFunctionsSuite. Added PySpark doctests for both functions. Regenerated sql-expression-schema.md.

Was this patch authored or co-authored using generative AI tooling?

No

[SreeramaYeshwanthGowd]

@cloud-fan Would you have a moment to review this when you get a chance? Thank you!

[SreeramaYeshwanthGowd]

Noting that the Base image build check is failing with a pull access denied error on docker.io/library/root:latest, unrelated to this PR.

[cloud-fan]

0 blocking, 1 non-blocking, 0 nits.
The implementation and API integration are sound; one small hot-path allocation can be avoided.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:3454: Avoid materializing a Java String for every decoded row; Commons Codec accepts the UTF8String bytes directly. -- see inline

Verification

I traced both public APIs through FunctionRegistry to the Catalyst RuntimeReplaceable expressions and compared their encode path with Base64. The added tests exercise codec semantics and each integration layer. I did not run tests locally.

[cloud-fan]

This conversion materializes a Java String for every non-null row. Commons Codec can decode the ASCII bytes directly, avoiding that hot-path allocation.

Suggested change

	codec.decode(input.toString)
	codec.decode(input.getBytes)

[cloud-fan]

Thanks, confirmed this is fixed in the current revision.

[SreeramaYeshwanthGowd]

@cloud-fan Thanks for the review and the suggestion, applied it exactly as proposed. Noting that the Base image build check is failing with a pull access denied error on docker.io/library/root:latest, unrelated to this PR.

[cloud-fan]

LGTM if CI is green

[SreeramaYeshwanthGowd]

@cloud-fan CI is green. Thans for the review.

[cloud-fan]

1 addressed, 0 remaining, 0 new.
0 blocking, 0 non-blocking, 0 nits.
The current implementation and API integration are sound; the prior review suggestion is addressed.

Verification

I traced both public API directions through FunctionRegistry to the Catalyst replacements and compared the encoding path with Base64. I also checked the current decoder uses UTF8String bytes directly and reviewed the added codec, SQL/DataFrame, and Connect coverage. I did not run tests locally.

[cloud-fan]

Merge Summary:

• merged into master 1684ada
• merged into branch-4.x 8176a10
• merged into branch-4.3 59d3291

Posted by merge_spark_pr.py