mySQL schema data model updates #19472

azhou-datadog · 2025-01-23T22:12:04Z

What does this PR do?

Updating the mySQL schema data model.

I've left columns and foreign keys as is, but partitions and indexes have been re-shaped. I've also removed a few columns that I don't find very useful, listed below. Please check me on these, here is the documentation for indexes and partitions

Partitions changes

Partitions have been re-shaped to avoid the pattern of a consumer of this dictionary needing to keep track of subpartition ordinal positions values to correlate a subpartition with its parent partition.
If subpartitions exist, the partition level fields data_length and table_rows are now aggregates of the same values in the subpartition dictionary. If there are no subpartitions, they behave as normal.
Removed partition_comment. We don’t include index comments, but if we want those, we can add those to indexes and keep this one too.
Removed index_lengths.
Removed data_free.
Removed max_data_lengths.
Removed tablespace_name.
Changed several string typed values to their actual underlying types (bools or ints).

Indexes changes

Indexes have similarly been re-shaped so an index built from multiple columns does not need to keep track of a seq_in_index value.
I’ve added the expression column to indexes. This is important for indexes that were built with Functional Key Parts. This was introduced in mySQL 8.0.13 (release notes, information_schema.statistics 8.0.13 docs)
Removed seq_in_index from indexes.
Changed several string typed values to their actual underlying types (bools or ints).
Collation should be associated per column, not the index as a whole.

Updated all unit tests to cover these changes.

Motivation

We want to display mySQL schemas in the schemas explorer page

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

mysql/datadog_checks/mysql/databases_data.py

codecov · 2025-01-23T22:58:45Z

Codecov Report

Attention: Patch coverage is 92.15686% with 4 lines in your changes missing coverage. Please review.

Project coverage is 88.07%. Comparing base (5150864) to head (9917e01).
Report is 18 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
activemq	`?`
cassandra	`?`
hive	`?`
hivemq	`?`
ignite	`?`
jboss_wildfly	`?`
kafka	`?`
mysql	`89.59% <92.15%> (?)`
presto	`?`
solr	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

github-actions · 2025-01-27T20:32:44Z

The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

github-actions · 2025-01-28T14:40:13Z

The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

azhou-datadog · 2025-01-28T14:53:27Z

mysql/datadog_checks/mysql/metadata.py

@@ -83,7 +83,6 @@ def __init__(self, check, config, connection_args):
        self._version_processed = False
        self._connection_args = connection_args
        self._db = None
-        self._check = check


deleting a duplicate, see L81

azhou-datadog · 2025-01-28T14:57:44Z

mysql/assets/configuration/spec.yaml

@@ -385,6 +385,7 @@ files:
            hidden: true
            description: |
              Configure collection of schemas (databases).
+              Only tables and schemas for which the user has been granted SELECT privileges are collected.


We will want to address this in a future PR, current best path forward seems to be setting up prepared statements that can query all schema info.

justiniso · 2025-01-28T15:04:53Z

mysql/datadog_checks/mysql/databases_data.py

+            index_data["cardinality"] = int(row["cardinality"])
+            index_data["index_type"] = str(row["index_type"])
+            index_data["non_unique"] = bool(row["non_unique"])
+            index_data["expression"] = str(row["expression"]).strip().lower() if row["expression"] else None


What's the purpose of .strip().lower() ?

Was thinking about normalizing some of these values at one point, but probably not necessary. Removed this

mysql/datadog_checks/mysql/queries.py

azhou-datadog requested review from a team as code owners January 23, 2025 22:12

temporal-github-worker-1 bot added agent/review-requested ecosystems/review-requested product/review-requested labels Jan 23, 2025

lu-zhengda reviewed Jan 23, 2025

View reviewed changes

mysql/datadog_checks/mysql/databases_data.py Outdated Show resolved Hide resolved

azhou-datadog added 3 commits January 27, 2025 17:07

logging testing

5a1af0c

new dictionary shape brainstorm

a76dc68

partitions portion of dictionary re-written

1d748f0

azhou-datadog force-pushed the allen.zhou/mysql-schema branch from ec6776b to 1d748f0 Compare January 27, 2025 17:07

azhou-datadog added 2 commits January 27, 2025 20:25

Indexes shape change and more testing

07febc0

lint and changelog

7879a8b

add subpart test, remove logging

b8cbb96

azhou-datadog added 3 commits January 28, 2025 09:40

Delete mysql/changelog.d/19472.changed

c6a448e

changelog

380cd29

delete log

63bced7

azhou-datadog changed the title ~~DRAFT: Allen.zhou/mysql schema~~ mySQL schema data model updates Jan 28, 2025

azhou-datadog commented Jan 28, 2025

View reviewed changes

justiniso reviewed Jan 28, 2025

View reviewed changes

azhou-datadog added 4 commits January 28, 2025 17:39

fix unit tests, review comments

7892162

linter!!!

c013d78

more unit tests fixes

8e4377e

mariadb version for collation

9917e01

justiniso approved these changes Jan 29, 2025

View reviewed changes

azhou-datadog added this pull request to the merge queue Jan 29, 2025

Merged via the queue into master with commit 3e22c06 Jan 29, 2025
34 checks passed

azhou-datadog deleted the allen.zhou/mysql-schema branch January 29, 2025 17:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mySQL schema data model updates #19472

mySQL schema data model updates #19472

azhou-datadog commented Jan 23, 2025 •

edited

Loading

codecov bot commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 27, 2025

github-actions bot commented Jan 28, 2025

azhou-datadog Jan 28, 2025

azhou-datadog Jan 28, 2025

justiniso Jan 28, 2025

azhou-datadog Jan 28, 2025

mySQL schema data model updates #19472

mySQL schema data model updates #19472

Conversation

azhou-datadog commented Jan 23, 2025 • edited Loading

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

codecov bot commented Jan 23, 2025 • edited Loading

Codecov Report

github-actions bot commented Jan 27, 2025

github-actions bot commented Jan 28, 2025

azhou-datadog Jan 28, 2025

Choose a reason for hiding this comment

azhou-datadog Jan 28, 2025

Choose a reason for hiding this comment

justiniso Jan 28, 2025

Choose a reason for hiding this comment

azhou-datadog Jan 28, 2025

Choose a reason for hiding this comment

azhou-datadog commented Jan 23, 2025 •

edited

Loading

codecov bot commented Jan 23, 2025 •

edited

Loading