Version 3.0.0 Proposal draft with sqlglot as the parsing library

[collerek]

---STILL WORK IN PROGRES---

Major rewrite of sql-metadata's parsing engine from token-based to sqlglot AST-based architecture (v3).

Refactor:

• Good starting point is ARCHITECTURE.md file
• Replaced token-based parser and sqlparse with sqlglot AST pipeline — raw SQL flows through SqlCleaner → DialectParser → sqlglot AST → specialized extractors, replacing manual tokenization
  and keyword matching
• Decomposed monolithic parser.py into focused extractor classes — ColumnExtractor, TableExtractor, NestedResolver, QueryTypeExtractor, DialectParser, SqlCleaner each own a
  single concern, composed by a thin Parser facade
• Added multi-dialect auto-detection — tries multiple sqlglot dialects (MySQL, TSQL, Spark, custom HashVarDialect) and picks the first non-degraded result
• Single-pass DFS column extraction — walks AST in arg_types key order preserving SQL text order, replacing multi-pass token scanning
• Removed legacy modules — token.py (token-based extraction), compat.py (v1 compatibility layer), .flake8 config

Feature:

• Added MERGE query type support

Admin:

• Added mypy with strict settings (disallow_untyped_defs, check_untyped_defs, warn_return_any) and fixed all type errors across 13 source files
• Added make type_check command and integrated mypy into CI workflow
• Switched linting/formatting fully to ruff — removed black workflow, black pre-install step, and pylint references from CI
• Added py.typed PEP 561 marker for downstream type checker support
• Added ARCHITECTURE.md with Mermaid diagrams, traced walkthroughs, and module deep dives, updated agents.md to reflect the rewrite

Resolved issues

• Resolves Issue with tables extraction #251
• Resolves Reusing an alias name on CTE's with the same column name produces an error #262
• Resolves Incorrect table names when using Presto UNNEST #284
• Resolves Parser crashes if column alias is the same as alias of the subquery #306
• Resolves Nested CTEs - Parser returns CTE as a table when there is nested CTE #314
• Resolves mssql sql query with top keyword problem #318
• Resolves complex sql no tables name #324
• Resolves When "FROM" multiple tables, the result will be different if the order is different? #335
• Resolves merge into syntax #354
• Resolves Update statement gets wrongly identified table #370
• Resolves Parser(sql).tables is listing variable name in SELECT-INTO as table name #397
• Resolves SEPARATOR being parsed as a column #400
• Resolves For Dateadd function , DD and WK keyword is been considered as column names #411
• Resolves Unable to parse uid and pad fields 【BUG】 #412
• Resolves with_name marked as table #413
• Resolves KeyError: 'JOIN' when parsing a query #424
• Resolves ERROR:Parser:Not supported query type: for 'create temporary table' #439
• Resolves Parser(stmt).tables return not expect table #446
• Resolves after closing parentheses parsing bug #447
• Resolves parser.columns lost the 2nd `` quoted columns with operation #448
• Resolves WHERE clause mistaken for table alias #451
• Resolves get_query_tables get not only table names but sometimes column names as well #457
• Resolves bug when extracting sub-queries #469
• Resolves BUG: The Hive ALTER statement was incorrectly parsed #495
• Resolves BUG: The Hive "INSERT OVERWRITE TABLE" statement was parsed incorrectly. #502
• Resolves The table_aliases method takes GROUP BY as an alias from CTE. #526
• Resolves Unmatched parentheses raises IndexError instead of an informative error message  #532
• Resolves determine last relevant keyword without considering ifnull() in the on clause #534
• Resolves "on" and "ON" are table aliases? #537
• Resolves Parser.tables returns wrong value when joined on datetrunc #555
• Resolves Parser.tables returns wrong values when join condition includes COALESCE #559
• Resolves Cannon find columns in 'SELECT COUNT(*) FROM examination WHERE 'Examination Date' > '1997-01-01' #578
• Resolves Does not find columns inside 'SUM-CASE-WHEN' pattern #579
• Resolves parser.tables and parser.columns parses empty when encountering quoted identifiers [BUG] #541
• Resolves [Feature Request] Extracting values from insert query for multi rows #558
• Resolves Parser.columns has bug #507
• Resolves Method parser.columns_dict do not take into account subqueries. #528
• Resolves Mypy: missing library stubs or py.typed marker #585
• Resolves Can't extract list of all columns with saving the order of them #468
• Resolves Feature Request: Window Function Column Resolution #421
• Resolves Library fails to parse inline function correctly while identifying columns #391
• Resolves Got uncorrect table names for complex sql #369
• Resolves ctas with column name dist/sort key breaks parser.tables #367
• Resolves How to properly extract subqueries? #365
• Resolves Some complex sql is not parsed correctly. #358
• Resolves error detect with block for bracketed select statement with as for columns #326
• Resolves Enhancement: GCP BigQuery UNNEST syntax #352
• Resolves Similar to issue #296 - When using alias.* in sub-query a column reference in parent query fails @ parser.columns #392

Disclaimer: The PR was written with a help of Claude althouth required a lot of manual fixes too ;)

[collerek]

@macbre started a proof of concept rewrite to replace quite stale and slow sqlparse with sqlglot as we had a convo with Toby some (quite long) time ago.

Let me know how you feel about that in general? (The idea - not the code details yet)

Seems we can close quite a lot of open issues, but replacing sqlparse was harder than I anticipated, as we do a lot of other things it seems.

Note it's still a work in progress. Was also working on it with Claude but it required quite some iterations and manual fixes anyway.

[macbre]

Sure, go for it 🚀 and welcome back!

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	mypy@​1.19.1
	librt@​0.8.1
	ruff@​0.11.13
	typing-extensions@​4.13.2 ⏵ 4.15.0
	ruff@​0.15.8
	sqlglot@​30.0.3
	sqlglot@​30.1.0

View full report