Azure AI Projects Python SDK (Azure#38338)

* First version of client with endpoint operations

* Make Endpoints sample work

* Create EndpointProperties class. Implement client.endpoints.get_default

* Support AzureAIClient.from_connection_string()

* Make default input argument for .endpoints.get

* Re-emit from latest TypeSpec. First attempt to support 3 different endpoints

* Fix construction of the three endpoints

* Re-emit from TypeSpec. Rename host_name to endpoint

* Bug fix

* Add methods ai_client.inference.get_chat_completions_client() and ai_client.inference.get_azure_openai_client()

* Adding sample and changing client name (Azure#37724)

* Add .inference.get_embeddings_client() method

* Jhakulin/azure ai client agents (Azure#37748)

* generated agents

* merge latest

* update

* add patches

* update

* new version generated

* fixes

* fix merge error

* update

* use assistant_id for now

* update

* Create seperate inference samples

* Move SASTokenCredential into .models

* Add more agents samples (Azure#37777)

* add more samples

* adjust sleep

* review updates

* Update _patch.py (Azure#37760)

* Online Evaluation changes (Azure#37794)

* Patch constructor for async AzureAIClient (Azure#37804)

* Users/singankit/evaluation add data mapping (Azure#37824)

* Update sample_evaluations.py

* Update sample_evaluations.py

* Fix .inference.get_azure_openai_client() for the async AzureAIClient (Azure#37828)

* migrate async, poll functions, and samples from assistant to agent (Azure#37831)

* migrate async, poll functions, and samples from assistant to agent

* migrate async, poll functions, and samples from assistant to agent

* resolved comments

* resolve comments

* using print instead of logging

* moved ai_client to main function

* Resolved comment

* Fixed samples

* [AI Client] adapt Sophia's assistant tests to client repo (Azure#37819)

* [AI Client] adapt Sophia's assistant tests to client repo

* get tests running and passing with pytest

* recordings pass and pushed

* review feedback

* Took out streaming from create_run, create_and_process_run, and submit_tool_outputs_to_run and declare separate functions for streaming (Azure#37851)

* Took out streaming from create_run, create_and_process_run, and submit_tool_outputs_to_run and declare separate functions for streaming

* Took out create_and_process_stream and update samples accordingly

* fixed samples

* Miscellaneous updates (Azure#37852)

* Jhakulin/azure ai client pylance (Azure#37887)

* Fixes to Pylance error

* fix sample

* update

* use file_upload_and_poll

* updates to async samples

* remove comment as we are using poll methods

* more fixes

* Use List instead of Sequence and fixed samples (Azure#37898)

* fixed warning for aio and get call function tools for stream within t… (Azure#37913)

* fixed warning for aio and get call function tools for stream within the SDK

* Fixed another warning

* Fixed return type

* Evaluation API Review Feedback (Azure#37915)

* Evaluation API Review Feedback

* add sample for evaluation schedule

---------

Co-authored-by: Sai Kothinti <[email protected]>

* Fixed poll while loop and sample (Azure#37922)

* Jhakulin/azure ai client agents updates2 (Azure#37937)

* minor fixes and updates

* update

* update

* create_vector_store_file_batch_and_poll + samples and update aio using the latest code from syn (Azure#37952)

* create_vector_store_file_batch_and_poll + samples and update aio using the latest code from sync

* Resolved comments

* Clean up on the Optional (Azure#37971)

* Switch from .endpoints to .connection (re-emit from latest TypeSpec)

* updates and minor fixes (Azure#37984)

* Clean up (Azure#37985)

* Rename connection string environment variables

Rename AI_CLIENT_CONNECTION_STRING to PROJECT_CONNECTION_STRING, to match how it's shown in Azure AI Studio.

Also rename AZURE_AI_CLIENT_AGENTS_CONNECTION_STRING (used in Agents tests) to PROJECT_CONNECTION_STRING_AGENTS_TESTS

* updated codeinterpreter attachment sample (Azure#37990)

* Filter parameters, returned by SSE event (Azure#37991)

* nirovins/filter_parameters

* Fixes

* Move utility functions out of class

* Evaluation data set id change revert (Azure#38007)

* Updates to .connection operators (Azure#38024)

- Re-emit from latest TypeSpec
- Updates to .connections operators to support the new preview version of REST API
- Add "id" on ConnectionProperties

* Users/singankit/ai project utils (Azure#38033)

* Sample upload file method on AI Client

* Updating sample evaluation

* Improving Evaluation user experience

* Adding pip install link for evaluation

* Update sample_evaluations.py

* Reverting connection changes

* Update _patch.py

* First emit of azure-ai-project (_patch.py, samples and test not yet copied)

* Copy _patch.py, samples, test, update names

* Fix wrong import

* M hietala/agent tools updates (Azure#38031)

* no need for function name separately in function definitions

* more changes related to not needing to specify function names separately

* adding capability to remove vector stores from file search tool

* adding remove file to code interpreter

* review comment related change

---------

Co-authored-by: Marko Hietala <[email protected]>

* add changes to latest typespec and ai-project changes (Azure#38051)

* Move agents test to agents folder (Azure#38055)

* Move agents test to agents folder

* Copy changes to project

* Prep for connection tests. (Azure#38060)

* Create azure-ai-projects folder

* Add Inference tests (Azure#38069)

* re-doing the agent toolu updates after package changes (Azure#38078)

Co-authored-by: Marko Hietala <[email protected]>

* Add .diagnostics.enable() method to fetch App Insights connection strings (Azure#38109)

* Users/singankit/evaluation async samples (Azure#38104)

* Adding async sample for evaluation

* Update sample_evaluations_async.py

* Fix sphinx docstring and cSpell (Azure#38125)

* Fix sphinx

* Fix spelling

* Fix

* Fix II

* Generate new code

* Fix test

* Fix

* Disable evaluation schedule and remove sampling rate (Azure#38161)

* sdk changes for online eval disable schedule and remove sampling rate

* change sample

* Add tests for Connection operations (Azure#38156)

* update to get file content (Azure#38165)

* Jhakulin/parse messages (Azure#38167)

* parse messages

* update

* add test file

* update

* use stream for get file content

* add get_file_content_stream

* update

* fix merge error

* review feedback

* more review comments

* more review comments

* Update README.md (Azure#38173)

* update to get file content (Azure#38192)

* Enable tracing (Azure#38185)

* added validation to tools and tools_resources, added update agent (Azure#38195)

* Rename `.diagnostics` to `.telemetry` (Azure#38209)

* Jhakulin/agents get file content async (Azure#38201)

* update to get file content

* agents async updates

* update

* fix await/pylance issue

* fix pylance error on AsyncToolSet

* return empty set if no tool resources for code interpreter

* added async toolset sample

* Fixed test (Azure#38222)

* Support the new unified models deployed to the AIServices connection

* Fixed validation to put AyncFunctionTool into ToolSet and FunctionTool into AsyncToolSet.   Introduce BaseToolset. (Azure#38244)

* adding ai agents instrumentor (Azure#38200)

* adding ai agents instrumentor

* streaming tracing sample and fixes to streaming tracing

* moved tracing to agents tracing folder

* adding support for tracing list_messages and async tracing sample

* removing some commented out code

* adding separate console and azure monitor tracing samples

* removing unused code

* removing more unused code

* renamed tracing to telemetry. removed throwing from instrument and uninstrument

* Update sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/agents/_utils.py

Co-authored-by: Liudmila Molkova <[email protected]>

* Update sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/agents/_ai_agents_instrumentor.py

Co-authored-by: Liudmila Molkova <[email protected]>

* Update sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/agents/_ai_agents_instrumentor.py

Co-authored-by: Liudmila Molkova <[email protected]>

* fix for streaming with eventhandler and toolset

* adding previous fix for async version also

* adding missing function

---------

Co-authored-by: Marko Hietala <[email protected]>
Co-authored-by: Liudmila Molkova <[email protected]>

* Jhakulin/user functions updates (Azure#38248)

* update to get file content

* agents async updates

* update

* fix await/pylance issue

* fix pylance error on AsyncToolSet

* return empty set if no tool resources for code interpreter

* added async toolset sample

* updates to user functions

* update

* fix the array type for parameters

* Connections & Inference test updates to support new unified models

* Improve tracing samples and add docs on telemetry.enable (Azure#38257)

* Improve tracig samples and add docs on telemetry.enable

* fix typo

* Jhakulin/user functions optional (Azure#38261)

* update to get file content

* agents async updates

* update

* fix await/pylance issue

* fix pylance error on AsyncToolSet

* return empty set if no tool resources for code interpreter

* added async toolset sample

* updates to user functions

* update

* fix the array type for parameters

* support for Optional

* update

* one more example with list of dicts

* adding content recording control to agents instrumentor api (Azure#38268)

Co-authored-by: Marko Hietala <[email protected]>

* Howie/save file test (Azure#38306)

* Added test for save_file

* resolved comment

* Mypy fixes for instrumentor (Azure#38302)

* Fixes

* Regenerate code

* Merge again

* Linter fixes

* Regenerate the code

* Fix mypy

* More mypy fixes

* Fix

* Fix handling of SAS token and fix mypy (Azure#38312)

* Fix handling of SAS token and fix mypy

* typo

* Fix also pylance issue

* Glharper/ai client new tools (Azure#38202)

* [AI Client] Add new search tools

* add hooks to update connection lists for bing search

* use updated typespec

* use ConnectionType and ConnectionsOperations

* endpointOperations -> ConnectionsOperations

* add AzureAISearch tool def

* adapt project with new tool changes

* Add new tool defs for bing and azure ai search to projects.models

* use service schema for BingGroundingTool

* add samples for new tools

* add sharepoint tool

* update prompt to use bing browsing

* add sharepoint_grounding discriminator

* code gen from merged typespec

* update to latest typespec

* review feedback

* add preview header to bing grounding and azure AI serach samples

* Use 'Union' insted of '|' to make it run on Pyton 3.8 and up (Azure#38344)

* Fix mypy issues and re generate the code. (Azure#38340)

* Add more mypy fixes

* Fix imports

* Fix

* Add safe conversion method

* Fix mypy

* Merge to main

* Fix mypy

* Fix unit test

* Sphinx fix

* Fix cspell

* Use DefaultAzureCredential from azure.identity.aio for async samples (Azure#38362)

* Rename AAD to ENTRA_ID (Azure#38363)

* added agents example section in readme and added tags to samples (Azure#38351)

* added agents example section in readme and added tags to samples

* Resolved comments

* Added more detail to readme

* Fix mypy and do not scan examples (Azure#38375)

* Changes in a package

* Fix

* Fix mypy

* fix mypy

* Fix

* Connections APIs throw instead of returning None if a connection name/type is not found (Azure#38383)

* Make some APIs throw instead of returning None. Re-try to get connection string

* Update README.md. Work in progress

* Fixes

* Add code interpreter file attachment sample and link to readme.   Add alot more detail to readme also. (Azure#38384)

* Fix broken async .telemetry.enable() method, when you don't pass in "destination" (Azure#38399)

* Fix README. Fix Optional[str]

* Fix async telemetry

* Sample for thread with resource and document it in readme (Azure#38393)

* Users/singankit/evaluation test (Azure#38402)

* Test file

* Adding fixture

* Adding evaluation e2e test

* Update azure_ai_projects_tests.env

* Update how the inference URL is constructed (Azure#38409)

* First

* Revert back this change

* Fix some cspell errors, and link errors (Azure#38420)

* cspell lint --config .vscode/cspell.json --no-summary "sdk/ai/azure-ai-projects/**/*"

* Resolved comments from Krista (Azure#38405)

* Address comments

* Run black

* Add support of Azure asset IDs for the vector store. (Azure#38066)

* Generate code

* Generate new code

* Expose file API and add the async test.

* Remove unnecessary change

* Add async samples

* Do not create ML Client in tests

* Better file retrieval

* Regenerate code

* Re generate the code

* Re generate code

* Fix samle name and make iterface more clear

* Add samples for attachment enterprise search

* Add unit tests for message attachment

* Finzlize sync tests.

* Add unit tests and use constants as a URI types

* Regenerate code

* Fix

* Generate new code

* Draft commit

* Add another unit test

* Fix and add test data

* Regenerate files

* Add more unit tests

* Re generate code

* Regenerate files

* Rename classes and regenerate the code

* Fixes

* Fix parameter name

* Fix file names

* Script to check sample names inside file comments.

* Fix PR comment by Krista on async credential in ConnectionProperties class (Azure#38424)

* Black

* User/singankit/evaluation spellcheck (Azure#38427)

* Spell check errors evaluation

* Removing test environment from evaluation schedule test

* Fix doc string errors

* Fix typing in samples (Azure#38354)

* Fix samples

* Re enable type checks and remove extra code

* Remove non needed code

* Remove non needed changes

* Remove non needed changes

* Fixes

* Return non optional client

* Remove extra line

* Fix file names

* Fixes

* Fix async file

* Fix mypy issues and suppress the ones related to typespec

* Fix cSpell

* Re generate code and fix unit tests

* Delete sdk/ai/azure-ai-projects/samples/config.json

* Fix typo

* Fix unit test

* Run "black" (again)

* Fix a few pyright errors, and address a comment in the Python PR (Azure#38435)

* Address some of Krista's PR comments (Azure#38444)

* sdk changes for name change for provisioning state and isEnabled online evaluation (Azure#38298)

* sdk changes for name change for provisioning state and isEnabled online evaluation

* add correct tsp-location.yaml

* azure-ai-projects: configure otel logging in telemetry.enable() helper (Azure#38434)

* Configure otel events in projects.telemetry.enable()

* [ai] fix(pylint): Resolve `W1309`, `W0107`, `W0707`, `R1705`, `W0401`, `R1714`, `W0611`, `W0404`, `W0201`, `W1201` and `W1203` (Azure#38447)

* style: Run isort

* fix(pylint): Resolve W1309(f-string-without-interpolation)

         From sdk/ai/azure-ai-projects, run:

             ruff check --select=F541 azure --fix

* fix(pylint): Resolve W0107(unnecessary-pass)

    From sdk/ai/azure-ai-projects, run

        ruff check --select=PIE790 --fix azure

* fix(pylint): Resolve W0707(raise-missing-from)

* fix(pylint): Resolve R1705(no-else-return)

    From azure/ai/azure-ai-projects, run

        ruff check --select=RET505 --fix azure

* fix(pylint): Resolve W0401(wildcard-import)

* fix(pylint): Resolve R1714(consider-using-in)

* fix(pylint): Resolve W0611(unused-import)

* fix(pylint): Resolve W0404(reimported)

* fix(pylint): Resolve W0201(attribute-defined-outside-init)

* fix(pylint): Resolve W1201(logging-not-lazy) and W1203(logging-fstring-interpolation)

* Add readme for get_message, save_file, get_file_content, tracing, Bing grounding, enterprise (Azure#38426)

* git Add readme for get_message, save_file, get_file_content, tracing

* Readme for enterprise search, code interpreter, attachment

* More documentation for tools

* Fixed create_thread documentation

* resolved comments

* Resolved comments

* Resolved comments from Krista

* More fixes to mypy (Azure#38446)

* Fix unit tests and some type issues.

* Fixes

* Fix mypy

* agents telemetry pyright fixes (Azure#38457)

Co-authored-by: Marko Hietala <[email protected]>

* Fix pyright for agents. (Azure#38464)

* Fix as if python version is 3.8

* Fixed

* Add custom config

* Move line, ignoring import

* fix pip install instruction (Azure#38480)

* [ai] fix(pylint): Resolve `C0301`, `C4739`, `C4740`, `C4741`, `C4742`, `C4743` for `azure-ai-projects` (Azure#38459)

* fix(pylint): Resolve C0301(line-too-long)

* fix(pylint): Resolve C4743(docstring-should-be-keyword)

* fix(pylint): Resolve C4739(docstring-missing-param)

* fix(pylint): Resolve C4740(docstring-missing-type)

* fix(pylint): Resolve C4741(docstring-missing-return)

* fix(pylint): Resolve C4742(docstring-missing-rtype)

* style: Run black

* Enable test recordings for tests\inference, tests\telemetry and tests\connection (Azure#38475)

* Update cspell.json

* Removed enterprise and add installation for tracing (Azure#38483)

* Removed enterprise and add installation for tracing

* clean up

* resolved comments

* Fix mypy errors (Azure#38487)

* Run black --config ../../../eng/black-pyproject.toml

* agents tracing pylint fixes (Azure#38496)

Co-authored-by: Marko Hietala <[email protected]>

* Remove premature adding of asserts.json file

* [ai] fix(pylint): Resolving remaining pylint errors (Azure#38500)

* fix(pylint): Resolve R1705(no-else-return)

* fix(pylint): Resolve C0207(use-maxsplit-arg)

* fix(pylint): Resolve C0412(ungrouped-imports)

* fix(pylint): Resolve W1401(anomalous-backslash-in-string)

* fix(pylint): Resolve W0105(pointless-string-statement)

* fix(pylint): Ignore W0221(arguments-differ)

         pylint has an open issue tracking false positives for
         arguments-differ with overloads in subclasses

* fix(pylint): Ignore W0718(broad-exception-caught)

* fix(pylint): Ignore E0401(import-error) and E0611(no-name-in-module)

* fix(pylint): Ignore C4748(client-accepts-api-verison-keyword)

    Currently, neither client classes supports overriding the api version

* fix(pylint): Ignore W0212(protected-access)

* fix(pylint): resolve W0707(raise-missing-from)

* fix(pylint): Ignore E1102(not-callable)

    pylint appears to fail to correctly infer that
    settings.tracing_implementation actually is callable

* fix(pylint): Ignore W0231(super-init-not-called)

* fix(pylint): Ignore W0613(unused-argument)

* fix(pylint): Ignore W0236(invalid-overriden-method)

* fix(pylint): Ignore R0914(too-many-locals)

* fix(pylint): Ignore R0902(too-many-instance-attributes)

* fix(pylint): Ignore R0915(too-many-statements)

* fix(pylint): Ignore R0911(too-many-return-statements)

* Disable all agents tests. Push recordings for all other tests to the assents folder (Azure#38481)

* run "black --config ../../../eng/black-pyproject.toml ."

* Fix pylint and new mypy issues (Azure#38503)

* Ignore most of pylint errors

* Add generated paths to pylint.rc

* Use backslash for consistency in pylintrc files

---------

Co-authored-by: Darren Cohen <[email protected]>

* Remove async from .telemetry.enable (Azure#38514)

* Fix pylint

* Rename `with_credentials` to `include_credentials` per apiview.dev review comment (Azure#38517)

* Try to fix test issues on Python 3.13. Run black

* Try again to fix Python 3.13 test failure

* Attempt to fix azure-ai-projects not showing up in the release pipeline

* Fix release date in CHANGELOG.md

* Fix CHANGELOG.md

---------

Co-authored-by: Ankit Singhal <[email protected]>
Co-authored-by: Jarno Hakulinen <[email protected]>
Co-authored-by: howieleung <[email protected]>
Co-authored-by: Glenn Harper <[email protected]>
Co-authored-by: Sai Kothinti <[email protected]>
Co-authored-by: nick863 <[email protected]>
Co-authored-by: M-Hietala <[email protected]>
Co-authored-by: Marko Hietala <[email protected]>
Co-authored-by: Liudmila Molkova <[email protected]>
Co-authored-by: kdestin <[email protected]>

Author: 11 people

.gitignore

@@ -158,3 +158,8 @@ sdk/cosmos/azure-cosmos/test/test_config.py
 
 # temporary folder to refresh SDK with cadl
 TempTypeSpecFiles/
+
+# Eclipse project files
+**/.project
+**/.pydevproject
+**/.settings

.vscode/cspell.json

@@ -33,6 +33,9 @@
     "sdk/batch/azure-batch/**",
     "sdk/ai/**/index/**",
     "sdk/ai/azure-ai-generative/tests/**",
+    "sdk/ai/azure-ai-projects/samples/agents/nifty_500_quarterly_results.csv",
+    "/sdk/ai/azure-ai-projects/samples/evaluations/async_samples/data/**",
+    "/sdk/ai/azure-ai-projects/samples/evaluations/data/**",
     "sdk/ai/azure-ai-resources/azure/ai/resources/_index/_langchain/vendor/**",
     "sdk/ai/azure-ai-resources/azure/ai/resources/_restclient/**",
     "sdk/cognitiveservices/azure-cognitiveservices-search-autosuggest/**",
@@ -1321,6 +1324,15 @@
         "smirnov"
       ]
     },
+    {
+      "filename": "sdk/ai/azure-ai-projects/**",
+      "words": [
+        "aiservices",
+        "OTEL",
+        "GENAI",
+        "fspath"
+      ]
+    },
     {
       "filename": "sdk/ai/azure-ai-inference/**",
       "words": [

eng/.docsettings.yml

@@ -15,6 +15,7 @@ omitted_paths:
   - sdk/ml/azure-ai-ml/tests/*
   - sdk/vision/azure-ai-vision-imageanalysis/tests/*
   - sdk/ai/azure-ai-inference/tests/*
+  - sdk/ai/azure-ai-projects/tests/*
   - sdk/storage/azure-storage-extensions/*
 
 language: python

eng/pylintrc

@@ -7,6 +7,7 @@ reports=no
 ignore-paths=
     azure\\mixedreality\\remoterendering\\_api_version.py,
     azure/mixedreality/remoterendering/_api_version.py,
+    (?:.*[/\\]|^)projects/(models/_models.py|_model_base.py|operations/_operations.py|aio/operations/_operations.py)$,
     # Exclude any path that contains the following directory names
     (?:.*[/\\]|^)(?:_vendor|_generated|_restclient|samples|examples|test|tests|doc|\.tox)(?:[/\\]|$)
 

pylintrc

@@ -7,6 +7,7 @@ reports=no
 ignore-paths=
     azure\\mixedreality\\remoterendering\\_api_version.py,
     azure/mixedreality/remoterendering/_api_version.py,
+    (?:.*[/\\]|^)projects/(models/_models.py|_model_base.py|operations/_operations.py|aio/operations/_operations.py)$,
     # Exclude any path that contains the following directory names
     (?:.*[/\\]|^)(?:_vendor|_generated|_restclient|samples|examples|test|tests|doc|\.tox)(?:[/\\]|$)
 

sdk/ai/azure-ai-projects/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Release History
+
+## 1.0.0b1 (2024-11-15)
+
+### Features Added
+
+First beta version

sdk/ai/azure-ai-projects/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) Microsoft Corporation.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sdk/ai/azure-ai-projects/MANIFEST.in

@@ -0,0 +1,7 @@
+include *.md
+include LICENSE
+include azure/ai/projects/py.typed
+recursive-include tests *.py
+recursive-include samples *.py *.md
+include azure/__init__.py
+include azure/ai/__init__.py