Skip to content

improve(bq_driver): SQLTables perf improvement - #1627

Open
Anshu6250 wants to merge 14 commits into
mainfrom
impr_sql_tables2
Open

improve(bq_driver): SQLTables perf improvement#1627
Anshu6250 wants to merge 14 commits into
mainfrom
impr_sql_tables2

Conversation

@Anshu6250

@Anshu6250 Anshu6250 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

perf result- link

Added google-cloud-cpp logs and can be verified with regex search "page_token: "[^"]" in file without max results there are 34 extra responses or can also search "next_page_token" which proves the hypothesis that is increasing the latency.
test_output_withoit_max_rsult.log
test_output_without_max_result.log

LiteralFromOdbcPattern method improves :- WildcardTableSearch majorly
With SQL_ATTR_METADATA_ID = SQL_FALSE, ODBC defines the catalog, schema and
table arguments as search patterns, and SQLTables treated all three that way
unconditionally. For a call that names an exact project and dataset, that meant
two wasted round trips before the useful one:

projects.list -> enumerate every visible project, regex-match to find the one named
datasets.list -> enumerate every dataset in it, regex-match to find the one named
tables.list -> the call we actually wanted

projects.list is the worst of the two: the API has no server-side name filter,
so it always returns everything just to select a name the caller already gave us.

LiteralFromOdbcPattern() checks whether a filter contains any unescaped % or
_. If it doesn't, the pattern can only ever match itself, so enumerate-then-match
is equivalent to using the name directly — the driver now does that and skips the
listing. This generalises the existing SQL_ATTR_METADATA_ID = SQL_TRUE fast path
to literal patterns.

No API call changed. No new endpoint, no changed request shape or parameters;
the surviving tables.list is identical, and the table-name filter is still matched
client-side as before. Two calls are simply no longer made. Escaped wildcards
(\_, \%) are unescaped and still treated as literals, so pattern semantics are
preserved.

EscapeOdbcPattern method improves :- FilterTablesOnDefaultDataset
When the application passes schema = NULL and the DSN sets
FilterTablesOnDefaultDataset=1, SQLTables substituted the configured dataset
name straight into dataset_filter — a variable everything downstream interprets
as an ODBC LIKE pattern. Since _ is a single-character wildcard, a configured
ODBC_TEST_DATASET compiles to ODBC.TEST.DATASET and also matches
ODBCxTESTyDATASET, so users could be shown tables from datasets they never
configured. BigQuery dataset names use _ as the conventional separator, so most
DSNs using this option are exposed.

EscapeOdbcPattern() escapes %, _ and \ at the substitution site so the name
matches only itself. It also lets the existing literal fast path skip
datasets.list for that call.

No API call changed — no new endpoint, no changed request shape; only how the
schema argument is interpreted before the existing calls are made.

@Anshu6250 Anshu6250 changed the title Impr sql tables2 improve(bq_driver): SQLTables perf improvement Aug 3, 2026
@shivamd-gpartner
shivamd-gpartner marked this pull request as ready for review August 5, 2026 15:21
@shivamd-gpartner
shivamd-gpartner requested a review from a team as a code owner August 5, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants