Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added new configuration to pass catalog name to metastore when creati… #24235

Open
wants to merge 21 commits into
base: master
Choose a base branch
from

Conversation

AnuragKDwivedi
Copy link

@AnuragKDwivedi AnuragKDwivedi commented Dec 10, 2024

Description

This PR introduces a new configuration that can be applied to Hive, Hudi, Delta, and Iceberg catalog properties. The configuration enables the catalog name to be passed to the metastore, significantly enhancing the metastore's capabilities for managing and organizing schemas and tables based on the catalog name.

By passing the catalog name, the metastore can now support unique schema creation under different catalogs, as it already recognizes the combination of catalog and schema as unique. Additionally, this change allows the metastore to filter schemas at the metastore layer itself, making schema management more efficient.

Motivation and Context

Previously, due to the absence of the catalog name in metastore interactions, all schemas were created under the default "hive" catalog. This limitation made it impossible for users to filter or retrieve schemas associated with a specific catalog. The metastore lacked the ability to distinguish between schemas created under different catalogs.

With this update:

  • Schemas can now be managed and organized by catalog, leveraging the metastore's existing support for catalog-schema uniqueness.
  • Users can create schemas with the same name under different catalogs, enabling better schema organization and reducing naming conflicts.
  • Schema filtering at the metastore layer becomes possible, providing more accurate and efficient responses to schema queries.
    This change addresses a long-standing limitation and significantly improves schema management in environments using Hive, Hudi, Delta, and Iceberg catalogs.

#22895

Impact

NA

Test Plan

CI passed

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.
== RELEASE NOTES ==
General Changes
Added configuration to pass catalog names to the metastore, enabling catalog-based schema management and filtering.

@AnuragKDwivedi AnuragKDwivedi force-pushed the catalog-name-to-metastore branch from 00027a7 to e72d7ff Compare December 12, 2024 05:28
@tdcmeehan tdcmeehan added the from:IBM PR from IBM label Dec 13, 2024
@prestodb-ci prestodb-ci requested review from a team, ShahimSharafudeen and Dilli-Babu-Godari and removed request for a team December 13, 2024 15:18
@prestodb-ci
Copy link

Saved that user @AnuragKDwivedi is from IBM

@steveburnett
Copy link
Contributor

Consider adding documentation for this configuration property. Perhaps in https://github.com/prestodb/presto/blob/master/presto-docs/src/main/sphinx/admin/properties.rst.

@AnuragKDwivedi AnuragKDwivedi force-pushed the catalog-name-to-metastore branch from 9b9b2d9 to c97e5bf Compare February 3, 2025 10:41
@AnuragKDwivedi AnuragKDwivedi marked this pull request as ready for review February 3, 2025 12:29
@AnuragKDwivedi AnuragKDwivedi marked this pull request as draft February 3, 2025 12:30
@steveburnett
Copy link
Contributor

New release note guidelines as of last week: PR #24354 automatically adds links to this PR to the release notes. Please remove the manual PR link in the following format from the release note entries for this PR.

:pr:`12345`

I have updated the Release Notes Guidelines to remove the examples of manually adding the PR link.

@AnuragKDwivedi AnuragKDwivedi marked this pull request as ready for review February 4, 2025 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
from:IBM PR from IBM
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants