Fixing context parsing issue with analyze_eval decorator (#26)

* Initial commit

Adding the Aimon Rely README, images, the postman collection, a simple client and examples.

A few small changes for error handling in the client and the example application.

Getting the Aimon API key from the streamlit app

updating README

Updating langchain example gif

Updating API endpoint

Adding V2 API with support for conciseness, completeness and toxicity checks (#1)

* Adding V2 API with support for conciseness, completeness and toxicity checks.

* Removing prints and updating config for the example application.

* Updating README

---------

Co-authored-by: Preetam Joshi <[email protected]>

Updating postman collection

Fixed the simple aimon client's handling of batch requests.

Updated postman collection. Added support for a user_query parameter in the input data dictionary.

Updating readme

Fixed bug in the example app

Uploading client code

Adding more convenience APIs

Fixing bug in create_dataset

Added Github actions config to publish to PyPI. Cleaned up dependencies and updated documentation.

Fixing langchain example

Fixing doc links

Formatting changes

Changes for aimon-rely

* Adding instruction adherence and hallucination v0.2 to the client

Updating git ignore

Adding more to gitignore

Removing .idea files

* Fixing doc string

* Updating documentation

* Updating Client to use V3 API

* Fixing test

* Updating tests

* Updating documentation in the client

* Adding .streamlit dir to .gitignore

* initial version of decorators for syntactic sugar

* A few more changes

* updating analyze and detect decorators

* Adding new notebooks

* Fixing bug in analyze decorator

* Updating Detect decorator to make it simpler. Adding Metaflow example. Adding documentation for the chatbot.

* fixing chatbot example

* Fixed issue in detect decorator. Improved code organization.

* fixed typo

* Updated the decorators with a more cleaner interface. Added a metaflow analyze example.

* Updated version

* Updated Notebook

* Fixing context parsing issue with analyze_eval decorator

---------

Co-authored-by: Preetam Joshi <[email protected]>

Author: pjoshi30

aimon/decorators/analyze.py

@@ -111,14 +111,14 @@ def _run_eval(self, func, args, kwargs):
         results = []
         for record in dataset_collection_records:
             result = func(record["context_docs"], record["user_query"], record["prompt"], *args, **kwargs)
-
+            _context = record['context_docs'] if isinstance(record['context_docs'], list) else [record['context_docs']]
             # TODO: Add instructions and config to the payload once supported
             payload = {
                 "application_id": self._am_app.id,
                 "version": self._am_app.version,
                 "prompt": record['prompt'] or "",
                 "user_query": record['user_query'] or "",
-                "context_docs": [d for d in record['context_docs']],
+                "context_docs": [d for d in _context],
                 "output": result,
                 "evaluation_id": self._eval.id,
                 "evaluation_run_id": eval_run.id,

examples/notebooks/aimon_decorators_langchain_summarization_0_5_0.ipynb

@@ -1,5 +1,24 @@
 {
  "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "d8495fd8-cd0a-406f-8f20-f6b66fd585fe",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[33mWARNING: You are using pip version 20.2.3; however, version 24.2 is available.\n",
+      "You should consider upgrading via the '/Users/preetamjoshi/projects/aimon/pj_aimon_rely/examples/chatbot/chtbot/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "!pip install langchain langchain-community --quiet"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 1,
@@ -18,7 +37,7 @@
    "outputs": [],
    "source": [
     "import os\n",
-    "openai_api_key = os.getenv(\"OPEN_AI_KEY\")"
+    "openai_api_key = os.getenv(\"OPENAI_API_KEY\")"
    ]
   },
   {
@@ -29,7 +48,7 @@
    "outputs": [],
    "source": [
     "analyze_eval = AnalyzeEval(\n",
-    "    Application(\"app_jun18_2024\"),\n",
+    "    Application(\"decorator_app_test\"),\n",
     "    Model(\"my_gpt4_model_fine_tuned\", \"GPT-4\"), \n",
     "    api_key=os.getenv(\"AIMON_API_KEY\"),\n",
     "    evaluation_name=\"simple_eval\",\n",
@@ -39,13 +58,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "id": "d5c3e5f0-7611-4e44-ba32-b721fba56218",
    "metadata": {},
    "outputs": [],
    "source": [
     "analyze_prod = AnalyzeProd(\n",
-    "    Application(\"llm_marketing_summarization_app\"), \n",
+    "    Application(\"decorator_app_prod\"), \n",
     "    Model(\"my_gpt4_model_v2\", \"GPT-4\"), \n",
     "    values_returned=[\"context\", \"generated_text\"],\n",
     "    api_key=os.getenv(\"AIMON_API_KEY\")\n",
@@ -64,7 +83,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 4,
    "id": "af98abfc-c04d-4cc7-ba3c-62f550de0c99",
    "metadata": {},
    "outputs": [],
@@ -97,17 +116,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 5,
    "id": "bcdddfa8-43c7-446a-9337-3ad0f16a015e",
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/Users/preetamjoshi/projects/aimon/pj_aimon_rely/examples/mflow/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The class `OpenAI` was deprecated in LangChain 0.0.10 and will be removed in 0.3.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAI`.\n",
+      "/Users/preetamjoshi/projects/aimon/pj_aimon_rely/examples/chatbot/chtbot/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The class `OpenAI` was deprecated in LangChain 0.0.10 and will be removed in 0.3.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAI`.\n",
       "  warn_deprecated(\n",
-      "/Users/preetamjoshi/projects/aimon/pj_aimon_rely/examples/mflow/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 0.3.0. Use invoke instead.\n",
+      "/Users/preetamjoshi/projects/aimon/pj_aimon_rely/examples/chatbot/chtbot/lib/python3.9/site-packages/langchain_core/_api/deprecation.py:139: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 0.3.0. Use invoke instead.\n",
       "  warn_deprecated(\n"
      ]
     }
@@ -118,15 +137,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 6,
    "id": "d6e94b08-0e2a-4860-a4f2-30d9ddea17af",
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "[(' Acme recently launched version 2.1 of their Python library, which has deep integrations with the Python ecosystem and has been proven to be beneficial for developers. This new version includes features like async support and improved error handling.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), ('\\n\\nTo configure the Acme python client, follow the official documentation which includes setting up environment variables and installing dependencies for both basic and advanced setups.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client is compatible with Python 3.6+ and multiple databases, including MySQL, PostgreSQL, and MongoDB. It is also suitable for cross-language projects with Node.js.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client may have installation, package conflicts, and connectivity issues. Troubleshooting involves checking the Python environment, dependencies, and log files, with specific error resolutions available in the online help section.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' Acme recently launched version 2.1 of their Python library, which has deep integrations with the Python ecosystem and has been proven to be beneficial for developers. This new version includes features like async support and improved error handling.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), ('\\n\\nTo configure the Acme python client, follow the official documentation which includes setting up environment variables and installing dependencies for both basic and advanced setups.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client is compatible with Python 3.6+ and multiple databases, including MySQL, PostgreSQL, and MongoDB. It is also suitable for cross-language projects with Node.js.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client may have installation, package conflicts, and connectivity issues. Troubleshooting involves checking the Python environment, dependencies, and log files, with specific error resolutions available in the online help section.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.'))]\n"
+      "[(' Acme recently launched version 2.1 of their Python library, which has deep integrations with the Python ecosystem and has been proven to be valuable for developers. This new version includes features like async support and improved error handling. Acme also supports Javascript and Java.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), ('\\n\\nTo configure the Acme python client, follow the official documentation which includes setting up environment variables and installing dependencies for both basic and advanced setups.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client is compatible with Python 3.6+ and multiple databases, including MySQL, PostgreSQL, and MongoDB. It is also suitable for cross-language projects with Node.js.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client may have installation, package conflicts, and connectivity issues. Troubleshooting involves checking the Python environment, dependencies, and log files, with specific error resolutions available in the online help section.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' Acme recently launched version 2.1 of their Python library, which has deep integrations with the Python ecosystem and has been proven to be valuable for developers. This new version includes features like async support and improved error handling. Acme also supports Javascript and Java.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), ('\\n\\nTo configure the Acme python client, follow the official documentation which includes setting up environment variables and installing dependencies for both basic and advanced setups.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client is compatible with Python 3.6+ and multiple databases, including MySQL, PostgreSQL, and MongoDB. It is also suitable for cross-language projects with Node.js.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.')), (' The Acme python client may have installation, package conflicts, and connectivity issues. Troubleshooting involves checking the Python environment, dependencies, and log files, with specific error resolutions available in the online help section.', AnalyzeCreateResponse(message='Data successfully sent to AIMon.'))]\n"
      ]
     }
    ],
@@ -146,7 +165,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 12,
    "id": "44438e7f-5652-46b5-9244-cbf2977d9d48",
    "metadata": {},
    "outputs": [],
@@ -174,7 +193,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 13,
    "id": "bb81a8ad-e10a-43f5-9682-844d9ab2ccb4",
    "metadata": {},
    "outputs": [],
@@ -194,7 +213,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 14,
    "id": "1e0dc9d7-e0b0-4a05-8967-0e06179486a3",
    "metadata": {},
    "outputs": [],
@@ -204,7 +223,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 15,
    "id": "5515eafd-8644-4792-a69e-23625a026c6c",
    "metadata": {},
    "outputs": [