[AQUA Telemetry] Update MD Tracking #1193

agrimk · 2025-05-22T13:24:23Z

https://jira.oci.oraclecorp.com/browse/ODSC-70841

github-actions · 2025-05-22T13:54:00Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-05-22T21:24:17Z

📌 Cov diff with main:

📌 Overall coverage:

VipulMascarenhas

added some comments.

Can we add some unit tests, and also set up pre-commit hook to format the code changes during commit? Also, can you also add some examples of logging events of both successful and unsuccessful in the description?

VipulMascarenhas · 2025-05-22T21:44:27Z

ads/aqua/modeldeployment/deployment.py

-        deployment_id = deployment.id
+
+
+        deployment_id = deployment.id()


id is a property, keep it as .id instead else it will result in TypeError

VipulMascarenhas · 2025-05-22T21:51:23Z

ads/common/work_request.py

@@ -38,6 +38,7 @@ def __init__(
        config: dict = None, 
        signer: Signer = None, 
        client_kwargs: dict = None, 
+        _error_message: str = None,


why do we need a parameter called _error_message here? We can have it inside init directly:

self._error_message = None

VipulMascarenhas · 2025-05-22T21:51:47Z

ads/aqua/modeldeployment/deployment.py

+
+
+    def get_deployment_status(self,model_deployment_id: str, work_request_id : str, model_type : str) :


add docstrings

VipulMascarenhas · 2025-05-22T22:02:05Z

ads/aqua/constants.py

@@ -54,6 +54,8 @@
 SERVICE_MANAGED_CONTAINER_URI_SCHEME = "dsmc://"
 SUPPORTED_FILE_FORMATS = ["jsonl"]
 MODEL_BY_REFERENCE_OSS_PATH_KEY = "artifact_location"
+DEFAULT_WAIT_TIME = 1200


since these constants are specific to model deployment status, we could have it in the ads.aqua.modeldeployment.constants.py.

VipulMascarenhas · 2025-05-22T22:02:49Z

ads/aqua/modeldeployment/deployment.py

@@ -80,6 +84,9 @@
 from ads.model.model_metadata import ModelCustomMetadataItem
 from ads.telemetry import telemetry

+THREAD_POOL_SIZE = 16


we can move this constant in ads.aqua.modeldeployment.constants.py

VipulMascarenhas · 2025-05-22T22:17:37Z

ads/aqua/modeldeployment/deployment.py

@@ -80,6 +84,9 @@
 from ads.model.model_metadata import ModelCustomMetadataItem
 from ads.telemetry import telemetry

+THREAD_POOL_SIZE = 16
+thread_pool = concurrent.futures.ThreadPoolExecutor(max_workers=THREAD_POOL_SIZE)


I think we should use the common telemetry or the common decorator threadpool instead of creating one here.

VipulMascarenhas · 2025-05-22T22:25:53Z

ads/aqua/modeldeployment/deployment.py

+                category=f"aqua/{model_type}/deployment/status",
+                action="FAILED",
+                detail="Error creating model deployment"


we should log the error message (_error_message) coming from work request here instead of a static message. This will be used to track the specific reasons why the deployment failed.

VipulMascarenhas · 2025-05-22T22:29:30Z

ads/aqua/modeldeployment/deployment.py

+            )
+
+        self.telemetry.record_event_async(


shouldn't this be within else block? We can use try-except-else here.

VipulMascarenhas · 2025-05-22T22:30:00Z

ads/aqua/modeldeployment/deployment.py

+                category=f"aqua/{model_type}/deployment/status",
+                action="SUCCEEDED",
+                detail=" Create model deployment successful", 


we can skip this detail, action "SUCCEEDED" implies the same thing.

…elerated-data-science into ODSC-70841_update_md_tracking

…o ODSC-70841_update_md_tracking

VipulMascarenhas

changes look good, added some minor comments. Can you run pre-commit hook to take care of some formatting issues?

pip install pre-commit
run pre-commit install

VipulMascarenhas · 2025-06-03T03:41:56Z

ads/aqua/modeldeployment/deployment.py

+            self.telemetry.record_event_async(
+                category=f"aqua/{model_type}/deployment/status",
+                action="FAILED",
+                detail=data_science_work_request._error_message


can _error_message be None for any reason? Might be good to do detail=data_science_work_request._error_message or UNKNOWN to avoid unforeseen issues in telemetry logging.

VipulMascarenhas · 2025-06-03T03:52:24Z

ads/common/work_request.py

@@ -78,6 +79,7 @@ def _sync(self):
        self._percentage= work_request.percent_complete
        self._status = work_request.status
        self._description = work_request_logs[-1].message if work_request_logs else "Processing"
+        if work_request.status == 'FAILED' : self._error_message = self.client.list_work_request_errors


might be good to show an example output for failed and successful MD in the PR description.

Also let's use ruff formatter to format the code.

mrDzurb · 2025-06-05T23:47:45Z

ads/aqua/modeldeployment/deployment.py

+            )
+        except Exception as e:
+            logger.error("Error while trying to create model deployment: " + str(e))
+            print("Error while trying to create model deployment: " + str(e))


Let's cleanup the print statement.

mrDzurb · 2025-06-05T23:49:03Z

ads/common/work_request.py

@@ -78,6 +79,7 @@ def _sync(self):
        self._percentage= work_request.percent_complete
        self._status = work_request.status
        self._description = work_request_logs[-1].message if work_request_logs else "Processing"
+        if work_request.status == 'FAILED' : self._error_message = self.client.list_work_request_errors


Also let's use ruff formatter to format the code.

mrDzurb · 2025-06-05T23:51:37Z

ads/telemetry/client.py

@@ -81,7 +81,8 @@ def record_event(
            # Here `endpoint`` is for debugging purpose
            # For some federated/domain users, the `endpoint` may not be a valid URL
            endpoint = f"{self.service_endpoint}/n/{self.namespace}/b/{self.bucket}/o/telemetry/{category}/{action}"
-            logger.debug(f"Sending telemetry to endpoint: {endpoint}")
+            logger.info(f"Sending telemetry to endpoint: {endpoint}")


let's use debug instead

github-actions · 2025-06-06T00:25:19Z

📌 Cov diff with main:

📌 Overall coverage:

…elerated-data-science into ODSC-70841_update_md_tracking

github-actions · 2025-06-09T13:34:44Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-06-10T06:49:04Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-06-18T11:36:55Z

📌 Cov diff with main:

📌 Overall coverage:

mrDzurb · 2025-06-18T16:29:50Z

ads/aqua/modeldeployment/constants.py

@@ -8,3 +8,6 @@

 This module contains constants used in Aqua Model Deployment.
 """
+
+DEFAULT_WAIT_TIME = 12000


This is in SEC?

I think there's an extra zero here, should have been 1200 :)

mrDzurb · 2025-06-18T16:32:26Z

ads/common/work_request.py

 # {l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, ' '{rate_fmt}{postfix}]
 # customize the bar format to remove the {n_fmt}/{total_fmt} from the right side
-DEFAULT_BAR_FORMAT = '{l_bar}{bar}| [{elapsed}<{remaining}, ' '{rate_fmt}{postfix}]'
+DEFAULT_BAR_FORMAT = "{l_bar}{bar}| [{elapsed}<{remaining}, " "{rate_fmt}{postfix}]"


Can't it be just

"{l_bar}{bar}| [{elapsed}<{remaining}, {rate_fmt}{postfix}]"

?

mrDzurb · 2025-06-18T16:34:57Z

Please, don't use Jira ticket as a reference. It would be better to add a description in the PR by itself.

github-actions · 2025-06-18T16:55:38Z

📌 Cov diff with main:

📌 Overall coverage:

VipulMascarenhas

the imports for MultiModelDeploymentConfigLoader seem to be incorrect, it should be from config_loader and not utils.

mrDzurb · 2025-06-18T17:33:00Z

@agrimk Could you aslo add a couple of the screenshots with the test results?

github-actions · 2025-06-18T18:16:37Z

📌 Cov diff with main:

📌 Overall coverage:

github-actions · 2025-06-18T19:07:24Z

📌 Cov diff with main:

📌 Overall coverage:

agrimk added 2 commits May 15, 2025 21:13

initial commit WIP

f6e3f00

ODSC-70841 Improve Model Deployment Status Tracking

619a925

agrimk requested review from darenr, mayoor, mrDzurb, VipulMascarenhas, qiuosier and ahosler as code owners May 22, 2025 13:24

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label May 22, 2025

Merge branch 'main' into ODSC-70841_update_md_tracking

2608622

VipulMascarenhas reviewed May 22, 2025

View reviewed changes

agrimk added 4 commits May 26, 2025 23:11

PR review comments

6917c98

Merge branch 'ODSC-70841_update_md_tracking' of github.com:oracle/acc…

075d714

…elerated-data-science into ODSC-70841_update_md_tracking

Merge branch 'main' of github.com:oracle/accelerated-data-science int…

bc41862

…o ODSC-70841_update_md_tracking

PR review comments

5e84eaf

VipulMascarenhas reviewed Jun 3, 2025

View reviewed changes

mrDzurb changed the title ~~Odsc 70841 update md tracking~~ [AQUA Telemetry] Update MD Tracking Jun 4, 2025

agrimk added 2 commits June 4, 2025 15:34

testing in INT

f3a9e9d

testing in INT

fc4c72a

mrDzurb reviewed Jun 5, 2025

View reviewed changes

Merge branch 'main' into ODSC-70841_update_md_tracking

fd09187

agrimk added 2 commits June 9, 2025 18:38

handling failure case and PR review comments

c0c072f

Merge branch 'ODSC-70841_update_md_tracking' of github.com:oracle/acc…

bf2953b

…elerated-data-science into ODSC-70841_update_md_tracking

adding env vars for tracking of SMC in telemetry

8b143ff

merge conflicts resolved

cabbfa4

moved wait work request to background thread

c2b8d41

Merge branch 'main' into ODSC-70841_update_md_tracking

a96bb82

mrDzurb previously approved these changes Jun 18, 2025

View reviewed changes

VipulMascarenhas approved these changes Jun 18, 2025

View reviewed changes

VipulMascarenhas requested changes Jun 18, 2025

View reviewed changes

agrimk added 2 commits June 18, 2025 23:19

corrected imports

a58ea43

pull from main branch

3938aac

agrimk dismissed mrDzurb’s stale review via 3938aac June 18, 2025 17:51

VipulMascarenhas and others added 3 commits June 18, 2025 11:18

Merge branch 'main' into ODSC-70841_update_md_tracking

8f2e0a0

fixed unit tests

4507162

merge of master

6eeb6cb

VipulMascarenhas approved these changes Jun 18, 2025

View reviewed changes

mrDzurb approved these changes Jun 18, 2025

View reviewed changes

VipulMascarenhas merged commit dee3591 into main Jun 18, 2025
24 checks passed

VipulMascarenhas deleted the ODSC-70841_update_md_tracking branch June 18, 2025 19:46

		deployment_id = deployment.id


		deployment_id = deployment.id()



		def get_deployment_status(self,model_deployment_id: str, work_request_id : str, model_type : str) :

[AQUA Telemetry] Update MD Tracking #1193

[AQUA Telemetry] Update MD Tracking #1193

Uh oh!

Conversation

agrimk commented May 22, 2025

Uh oh!

github-actions bot commented May 22, 2025

Uh oh!

github-actions bot commented May 22, 2025

Uh oh!

VipulMascarenhas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VipulMascarenhas May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VipulMascarenhas May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VipulMascarenhas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 6, 2025

Uh oh!

github-actions bot commented Jun 9, 2025

Uh oh!

github-actions bot commented Jun 10, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrDzurb commented Jun 18, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

VipulMascarenhas left a comment

Choose a reason for hiding this comment

Uh oh!

mrDzurb commented Jun 18, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

VipulMascarenhas left a comment •

edited

Loading

VipulMascarenhas May 22, 2025 •

edited

Loading

VipulMascarenhas May 22, 2025 •

edited

Loading