Skip to content

Fix MySQL DELETE using CONCAT_WS preventing index usage#5756

Open
hashwnath wants to merge 1 commit intoSQLMesh:mainfrom
hashwnath:fix/5711-mysql-delete-concat-ws
Open

Fix MySQL DELETE using CONCAT_WS preventing index usage#5756
hashwnath wants to merge 1 commit intoSQLMesh:mainfrom
hashwnath:fix/5711-mysql-delete-concat-ws

Conversation

@hashwnath
Copy link
Copy Markdown

Summary

Fixes #5711

  • Override _replace_by_key in MySQLEngineAdapter to use a JOIN-based DELETE for composite keys instead of CONCAT_WS('__SQLMESH_DELIM__', ...) which prevents MySQL/MariaDB from using indexes
  • Single-key DELETEs continue to use the existing efficient IN-based approach since indexes work fine for single columns
  • Add tests verifying both composite key (JOIN-based) and single key (IN-based) DELETE behavior

Problem

When using INCREMENTAL_BY_UNIQUE_KEY with MySQL/MariaDB and composite keys, SQLMesh generates:

DELETE FROM `target_table`
WHERE CONCAT_WS('__SQLMESH_DELIM__', `col1`, `col2`)
IN (SELECT CONCAT_WS('__SQLMESH_DELIM__', `col1`, `col2`) FROM `temp_table`)

MySQL cannot use indexes on CONCAT_WS() expressions, causing full table scans that lead to multi-hour query times and connection timeouts on larger tables.

Fix

For composite keys, generate a JOIN-based DELETE that allows MySQL to use existing indexes:

DELETE `_target`
FROM `target_table` AS `_target`
INNER JOIN `temp_table` AS `_temp`
ON `_target`.`col1` = `_temp`.`col1`
AND `_target`.`col2` = `_temp`.`col2`

This changes the EXPLAIN plan from type=ALL (full scan) to type=ref (index lookup), reducing query times from hours to seconds.

Test plan

  • New test test_replace_by_key_composite_uses_join_delete verifies JOIN-based DELETE for composite keys
  • New test test_replace_by_key_single_key_uses_in verifies single-key DELETEs still use the IN-based approach
  • Existing MySQL adapter tests pass
  • Existing mixin tests pass
  • Existing base engine adapter tests pass

Override _replace_by_key in MySQLEngineAdapter to use a JOIN-based
DELETE for composite keys instead of CONCAT_WS('__SQLMESH_DELIM__', ...).

MySQL/MariaDB cannot use indexes on CONCAT_WS expressions, causing full
table scans on every incremental run. The JOIN-based approach allows the
engine to use existing composite indexes, reducing query times from hours
to seconds on large tables.

Single-key DELETEs continue to use the efficient IN-based approach.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MySQL/MariaDB: INCREMENTAL_BY_UNIQUE_KEY DELETE uses CONCAT_WS which prevents index usage → full table scan / timeout

1 participant