Skip to content

Add global resource support #4003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add global resource support #4003

wants to merge 2 commits into from

Conversation

zirui-song-18
Copy link

Description

Overview

Global resource is resources that can be accessed by any tenant, this PR is to support global resource including connector, model and agent.

Changes

  • Multi-tenant encryption: Added logic to differentiate between global and tenant-specific resources for credential decryption
  • Remote metadata integration: Migrated config retrieval to use SDK client with DynamoDB support
  • Configuration enhancement: Added global tenant ID setting for cross-tenant resource management

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 25, 2025 07:29 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 25, 2025 07:29 — with GitHub Actions Error
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 25, 2025 07:29 — with GitHub Actions Error
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 25, 2025 07:29 — with GitHub Actions Failure
xinyual
xinyual reviewed Jul 29, 2025
View reviewed changes
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 30, 2025 00:27 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 30, 2025 00:27 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 30, 2025 00:27 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval July 30, 2025 00:27 — with GitHub Actions Failure
@dhrubo-os dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval August 1, 2025 05:49 — with GitHub Actions Failure
@dhrubo-os dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval August 1, 2025 05:49 — with GitHub Actions Failure
@dhrubo-os dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval August 1, 2025 05:49 — with GitHub Actions Failure
@dhrubo-os dhrubo-os had a problem deploying to ml-commons-cicd-env-require-approval August 1, 2025 05:49 — with GitHub Actions Failure
dhrubo-os
dhrubo-os reviewed Aug 1, 2025
View reviewed changes
ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/remote/RemoteModel.java Outdated
@Override
public void initModel(MLModel model, Map<String, Object> params, Encryptor encryptor) {
try {
SdkClient sdkClient = (SdkClient) params.get(SDK_CLIENT);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how are you getting this param?

@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 9, 2025 07:11 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 16, 2025 02:34 — with GitHub Actions Error
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 16, 2025 02:34 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 16, 2025 02:34 — with GitHub Actions Error
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 16, 2025 02:34 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 17, 2025 02:37 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 17, 2025 02:37 — with GitHub Actions Failure
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 17, 2025 02:37 — with GitHub Actions Error
@zirui-song-18 zirui-song-18 had a problem deploying to ml-commons-cicd-env-require-approval September 17, 2025 02:37 — with GitHub Actions Error
@zane-neo zane-neo force-pushed the global_resource_support branch from d7cb50f to e329bd3 Compare September 17, 2025 07:30
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:31 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:31 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:31 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:31 — with GitHub Actions Waiting
@zane-neo zane-neo force-pushed the global_resource_support branch from e329bd3 to d0c1266 Compare September 17, 2025 07:35
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:36 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:36 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:36 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 17, 2025 07:36 — with GitHub Actions Waiting
@zane-neo zane-neo force-pushed the global_resource_support branch from d0c1266 to f21bde8 Compare September 18, 2025 09:04
@zane-neo zane-neo had a problem deploying to ml-commons-cicd-env-require-approval September 18, 2025 09:05 — with GitHub Actions Failure
@zane-neo zane-neo had a problem deploying to ml-commons-cicd-env-require-approval September 18, 2025 09:05 — with GitHub Actions Error
@zane-neo zane-neo had a problem deploying to ml-commons-cicd-env-require-approval September 18, 2025 09:05 — with GitHub Actions Failure
@zane-neo zane-neo had a problem deploying to ml-commons-cicd-env-require-approval September 18, 2025 09:05 — with GitHub Actions Error
zirui-song-18 and others added 2 commits September 19, 2025 14:09
@zirui-song-18 @zane-neo
feat: Add global resource support for remote metadata with DynamoDB i…
d56b68a 
…ntegration

- Add DynamoDB client dependency for remote metadata storage
- Implement tenant-aware encryption for global vs tenant-specific resources
- Migrate GetConfigTransportAction to use SDK client instead of direct ES client
- Add REMOTE_METADATA_GLOBAL_TENANT_ID setting support
- Force AWS SDK version alignment for dependency consistency

Signed-off-by: Zirui Song <[email protected]>
@zane-neo
Fix config get transport action test failures
daeeef6 
Signed-off-by: zane-neo <[email protected]>
@zane-neo zane-neo force-pushed the global_resource_support branch from f21bde8 to daeeef6 Compare September 19, 2025 06:10
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 19, 2025 06:11 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 19, 2025 06:11 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 19, 2025 06:11 — with GitHub Actions Waiting
@zane-neo zane-neo requested a deployment to ml-commons-cicd-env-require-approval September 19, 2025 06:11 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xinyual xinyual xinyual left review comments

@dhrubo-os dhrubo-os dhrubo-os left review comments

@b4sjoo b4sjoo Awaiting requested review from b4sjoo b4sjoo is a code owner

@mingshl mingshl Awaiting requested review from mingshl mingshl is a code owner

@jngz-es jngz-es Awaiting requested review from jngz-es jngz-es is a code owner

@model-collapse model-collapse Awaiting requested review from model-collapse model-collapse is a code owner

@rbhavna rbhavna Awaiting requested review from rbhavna rbhavna is a code owner

@ylwu-amzn ylwu-amzn Awaiting requested review from ylwu-amzn ylwu-amzn is a code owner

@zane-neo zane-neo Awaiting requested review from zane-neo zane-neo is a code owner

@Zhangxunmt Zhangxunmt Awaiting requested review from Zhangxunmt Zhangxunmt is a code owner

@austintlee austintlee Awaiting requested review from austintlee austintlee is a code owner

@HenryL27 HenryL27 Awaiting requested review from HenryL27 HenryL27 is a code owner

@sam-herman sam-herman Awaiting requested review from sam-herman sam-herman is a code owner

@pyek-bot pyek-bot Awaiting requested review from pyek-bot pyek-bot is a code owner

+1 more reviewer

@will-hwang will-hwang will-hwang left review comments

Reviewers whose approvals may not affect merge requirements

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
infra
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zirui-song-18 @dhrubo-os @will-hwang @xinyual @zane-neo