Open
Conversation
YoYoJa
requested review from
bobogogo1990,
dargilco,
glharper,
howieleung,
kingernupur,
nick863,
trangevi and
trrwilson
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates the Azure AI Projects evaluation sample code to reflect the new initialization parameter name used by evaluators.
Changes:
- Replaces
initialization_parameters.deployment_namewithinitialization_parameters.modelacross evaluation samples - Updates both scheduled and inline/dataset-based evaluation examples to use the new parameter key
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/ai/azure-ai-projects/samples/evaluations/sample_scheduled_evaluations.py | Updates scheduled evaluation sample to use model instead of deployment_name in evaluator init params. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_evaluations_builtin_with_inline_data_oai.py | Updates OAI inline-data sample evaluator init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_evaluations_builtin_with_inline_data.py | Updates inline-data sample evaluator init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_evaluations_builtin_with_dataset_id.py | Updates dataset-id sample evaluator init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_evaluations_ai_assisted.py | Updates AI-assisted evaluation sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_eval_catalog_prompt_based_evaluators.py | Updates catalog prompt-based evaluator sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_agent_response_evaluation_with_function_tool.py | Updates agent response evaluation sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/sample_agent_evaluation.py | Updates agent evaluation sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_tool_selection.py | Updates agentic tool selection sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_tool_output_utilization.py | Updates agentic tool output utilization sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_tool_input_accuracy.py | Updates agentic tool input accuracy sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_tool_call_success.py | Updates agentic tool call success sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_tool_call_accuracy.py | Updates agentic tool call accuracy sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_task_completion.py | Updates agentic task completion sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_task_adherence.py | Updates agentic task adherence sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_response_completeness.py | Updates agentic response completeness sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_relevance.py | Updates agentic relevance sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_intent_resolution.py | Updates agentic intent resolution sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_groundedness.py | Updates agentic groundedness sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_fluency.py | Updates agentic fluency sample init params to use model. |
| sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_coherence.py | Updates agentic coherence sample init params to use model. |
| "evaluator_name": "builtin.violence", | ||
| "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}, | ||
| "initialization_parameters": {"deployment_name": "{{aoai_deployment_and_model}}"}, | ||
| "initialization_parameters": {"model": "{{aoai_deployment_and_model}}"}, |
There was a problem hiding this comment.
The PR description is still the default template text (asks to 'add an informative description', checklist unchecked, no linked issue/spec PR). Please update the PR description to briefly explain the deployment_name -> model parameter change, and link any relevant API/spec change or tracking issue so reviewers can validate the samples against the intended API behavior.
| "evaluator_name": "builtin.violence", | ||
| "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}, | ||
| "initialization_parameters": {"deployment_name": f"{model_deployment_name}"}, | ||
| "initialization_parameters": {"model": f"{model_deployment_name}"}, |
There was a problem hiding this comment.
The initialization parameter key is now model, but the value is still sourced from a variable named model_deployment_name, which is potentially confusing to sample readers (is this a model name or a deployment name?). Consider renaming the variable (and related comments/strings) to match the new model terminology, or add a short inline comment clarifying what model expects here.
| "evaluator_name": "builtin.violence", | ||
| "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}, | ||
| "initialization_parameters": {"deployment_name": f"{model_deployment_name}"}, | ||
| "initialization_parameters": {"model": f"{model_deployment_name}"}, |
There was a problem hiding this comment.
The f-string f\"{model_deployment_name}\" is redundant when interpolating an existing string variable. Using model_deployment_name directly makes the sample slightly clearer and avoids implying formatting is needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines