Skip to content

[FLINK-41433][docs] Add pages for the regular expression, cast, conditional, window and utility functions - #29864

Open
raminqaf wants to merge 8 commits into
apache:masterfrom
raminqaf:FLINK-41433
Open

raminqaf wants to merge 8 commits into
apache:masterfrom
raminqaf:FLINK-41433

Conversation

@raminqaf

@raminqaf raminqaf commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Step 2 of FLINK-23804: every function of five categories gets a page of its own, generated from YAML. A page has the SQL, Java and Python syntax, the arguments, what it returns and when it returns NULL, usage notes, and examples. Each function was rechecked against its implementation, and every example result comes from running its SQL. The recheck changed what the docs say, for example:

Function Old docs Master
REGEXP_REPLACE Says nothing about the replacement $1 is inserted as written. The replacement can't refer to groups.
REGEXP_EXTRACT_ALL extractIndex defaults to 1 So a regex without groups returns NULL unless you pass 0
RANK "one plus the number of rows preceding or equal to the current row" One plus the rows before it. Peers aren't counted.
TRY_CAST COALESCE(TRY_CAST('non-number' AS INT), 0) "returns 0 of type INT" The type is INT NOT NULL
LEAD No mention of streaming Not supported in streaming mode

Brief change log

  • Bump Hugo from 0.124.1 to 0.126.3. The pages come from a content adapter, which needs Hugo 0.126. The docs CI script passes --logLevel info instead of -v, which 0.126.3 logs as a deprecation error.
  • Generate one page per data/functions/<category>/<name>.yml, in English and Chinese. A page starts with its category, a one-line summary and related functions, followed by the same sections: syntax, arguments, returns, usage notes and examples. A render hook for SQL code blocks highlights the page's function in its syntax and examples.
  • In the category tables, a converted row shows the function's name instead of its signature, links to its page and shows the page's summary. The two CASE rows show once, and the rows are in alphabetical order, as in the sidebar. The A-Z page links converted rows to their page, and a category in the sidebar expands to its function pages.
  • Use the Flink colors for code blocks and the sidebar. Keywords and SQL types are bold purple, strings dark orange and numbers black. On a function page, the page's function is bold dark red. The SQL render hook gives every SQL type name the same class, because the highlighter tags INT, TIMESTAMP and STRING in three different ways. The colors apply to every code block on the site.
  • Add 33 pages: Regular Expressions 7, Cast 2, Conditional 7, Window 9, Utility 8. Every row of these categories now points at a page.
  • Chinese pages show the English sections. Where a row had a Chinese description, its first sentence becomes the summary.
  • Two Table API-only rows lose their Scala-only form: BOOLEAN.? folds into the IF row as BOOLEAN.then, and NUMERIC.rows becomes rowInterval(LONG).
  • Add a guide for function pages to docs/README.md, including what a page leaves out, and point AGENTS.md step 8 at it.

Verifying this change

This change is documentation only.

  • Every SQL example ran in batch and streaming mode on master with a local runner, which isn't part of this PR. The results on the pages are pasted from those runs.
  • A second review per category checked each statement against the type strategies, the runtime code and the tests.
  • Hugo 0.126.3 builds both languages without warnings. A link check finds no new broken internal links. The A-Z page still lists 241 functions in English and 238 in Chinese, as on master, and 15 Table API-only entries instead of 16.
  • Every commit builds on its own.

The recheck found bugs in the code, which this PR doesn't fix. The pages describe the intended behavior, and I'll file the bugs separately.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.322 (Claude Opus 5.5)

@flinkbot

flinkbot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf
raminqaf force-pushed the FLINK-41433 branch 2 times, most recently from 2b03c16 to 3b4c609 Compare November 12, 2026 10:31
Comment thread docs/data/functions/cast/try_cast.yml Resolved
Comment thread docs/data/functions/cast/cast.yml Resolved
Comment thread docs/data/functions/conditional/if.yml Resolved
Comment thread docs/data/functions/window/ntile.yml Resolved
The built-in function pages are generated by a content adapter, which needs Hugo 0.126. Hugo 0.126.3 builds the current docs with the same page content. Later versions need changes to the templates and the config, for example because they removed resources.ToCSS.

The README's Docker command now uses an image with the same Hugo version. The docs CI script passes --logLevel info instead of -v, which Hugo 0.126 logs as a deprecation error and 0.127 removes.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
A content adapter adds one page for each data/functions/<category>/<name>.yml, in English and Chinese. Every page starts with its category, a one-line summary and related functions, followed by the same sections: syntax, arguments, returns, usage notes and examples. A file with the same path under data/functions_zh/ replaces fields of the Chinese page.

A render hook for SQL code blocks highlights the function of the page in its syntax and examples. SQL blocks on other pages render as before.

A row in sql_functions.yml with a page field links to its page from the category table and from the A-Z list. The category table shows the function's name instead of its signature, and the page's summary. Rows for two forms of one function, such as CASE, show once. The category sections in the sidebar expand to their function pages. A collapsible section shows its arrow only if it has pages to show.

The README describes how to add a function page, and the built-in function guide in AGENTS.md points to it.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- REGEXP_REPLACE inserts the replacement as written, so $1 doesn't refer to a group.
- Only REGEXP, REGEXP_EXTRACT and REGEXP_REPLACE check a literal regex at planning time. The other functions return NULL for a regex that doesn't compile.
- REGEXP_EXTRACT_ALL defaults to group 1, so a regex without groups returns NULL unless the index is 0.
- REGEXP_INSTR counts UTF-16 code units.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- CASE, IF, COALESCE and IFNULL evaluate only the branch or argument they need.
- The result type of each function, and the types that IF, GREATEST and LEAST accept.
- The Scala-only BOOLEAN.? row moves into the IF row, whose Table API call is BOOLEAN.then(...).

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- A type pair that the casting table doesn't allow fails at planning time, for CAST and TRY_CAST alike.
- TRY_CAST always returns a nullable type, so COALESCE(TRY_CAST(...), 0) is NOT NULL.
- Casts between numeric types wrap around or saturate instead of failing.
- CHAR(n) and BINARY(n) targets cut and pad, VARCHAR(n) and VARBINARY(n) only cut.

The see-also link to TYPEOF points at the Utility category until TYPEOF has a page.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- UUID returns CHAR(36). UUID_V4 and UUID_V7 return the UUID type.
- CURRENT_DATABASE is read once per query.
- TYPEOF evaluates its input, and force_serializable must be a literal.
- DESCRIPTOR is allowed in SQL only as an argument of a table function.
- The Table API-only callSql and as get pages with Java and Python examples.

The see-also link from CAST to TYPEOF now points at the new page.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- Which functions run in streaming mode. ROW_NUMBER and LAG do in some query shapes, the others are supported in batch mode only.
- RANK is one plus the number of rows before the current row. The old row also counted its peers.
- The result types, and the arguments that LEAD, LAG and NTILE accept.
- The Scala-only NUMERIC.rows row becomes rowInterval(LONG), with a page for the Java and Python call.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.321 (Claude Opus 5.5)
Code blocks use the colors of the Flink logo instead of the GitHub style. Keywords and SQL types are bold purple, strings dark orange and numbers black, like names. On a built-in function page, the page's function is bold dark red, so it stands out from the keywords.

The SQL render hook gives every SQL type name the same class. The highlighter tags INT, TIMESTAMP and STRING in three different ways, so the same type showed in different colors.

The sidebar marks the current page in purple with a bar in the light purple of the logo.

Generated-by: Claude Code 2.1.322 (Claude Opus 5.5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants