[JS] fix: CardGazer sample (#2059)

## Linked issues

#2039
#minor


<img width="935" alt="image"
src="https://github.com/user-attachments/assets/0018973c-9751-46d5-abd1-b6cbe889d9cc">

## Details

- `OpenAIModel.ts` docstring cleanup
- `PromptManager` - change order of tools and user input
- `UserInputMessage` - don't send messages if messages are empty.
- Update cardgazer to use 4o-mini and tools augmentation
- Update CardGazer sample to improve experience.
- Add note in README for Bot SSO sample


## Attestation Checklist

- [x] My code follows the style guidelines of this project

- I have checked for/fixed spelling, linting, and other errors
- I have commented my code for clarity
- I have made corresponding changes to the documentation (updating the
doc strings in the code is sufficient)
- My changes generate no new warnings
- I have added tests that validates my changes, and provides sufficient
test coverage. I have tested with:
  - Local testing
  - E2E testing in Teams
- New and existing unit tests pass locally with my changes

### Additional information

> Feel free to add other relevant information below

---------

Co-authored-by: Corina Gum <>

Author: corinagum

js/packages/teams-ai/src/prompts/PromptManager.ts

@@ -335,17 +335,17 @@ export class PromptManager implements PromptFunctions {
                 );
             }
 
+            if (template.config.augmentation && template.config.augmentation.augmentation_type === 'tools') {
+                const includeHistory: boolean = template.config.completion.include_history;
+                const historyVariable = includeHistory ? `conversation.${name}_history` : 'temp.${name}_history';
+                sections.push(new ActionOutputMessage(historyVariable));
+            }
             // Include user input
             if (template.config.completion.include_images) {
                 sections.push(new UserInputMessage(this.options.max_input_tokens));
             } else if (template.config.completion.include_input) {
                 sections.push(new UserMessage('{{$temp.input}}', this.options.max_input_tokens));
             }
-            if (template.config.augmentation && template.config.augmentation.augmentation_type === 'tools') {
-                const includeHistory: boolean = template.config.completion.include_history;
-                const historyVariable = includeHistory ? `conversation.${name}_history` : 'temp.${name}_history';
-                sections.push(new ActionOutputMessage(historyVariable));
-            }
 
             // Create prompt
             template.prompt = new Prompt(sections);

js/packages/teams-ai/src/prompts/UserInputMessage.ts

@@ -81,7 +81,9 @@ export class UserInputMessage extends PromptSectionBase {
         const images = inputFiles.filter((f) => f.contentType.startsWith('image/'));
         for (const image of images) {
             // Check for budget to add image
-            // TODO: This accounts for low detail images but not high detail images.
+            // This accounts for low detail images but not high detail images.
+            // https://platform.openai.com/docs/guides/vision
+            // low res mode defaults to a 512x512px image which is budgeted at 85 tokens.
             // Additional work is needed to account for high detail images.
             if (budget < 85) {
                 break;
@@ -99,7 +101,11 @@ export class UserInputMessage extends PromptSectionBase {
             budget -= 85;
         }
 
+        const output = [];
+        if (message.content!.length > 0) {
+            output.push(message);
+        }
         // Return output
-        return { output: [message], length, tooLong: false };
+        return { output, length, tooLong: false };
     }
 }

js/samples/03.ai-concepts/c.actionMapping-lightBot/src/prompts/tools/config.json

@@ -3,7 +3,7 @@
     "description": "A bot that can turn the lights on and off",
     "type": "completion",
     "completion": {
-        "model": "o1-preview",
+        "model": "gpt-4o",
         "completion_type": "chat",
         "include_history": true,
         "include_input": true,
@@ -14,6 +14,6 @@
         "stop_sequences": []
     },
     "augmentation": {
-        "augmentation_type": "monologue"
+        "augmentation_type": "tools"
     }
 }

js/samples/04.ai-apps/c.vision-cardGazer/README.md

@@ -18,7 +18,7 @@ _Table of contents_
 
 ## Interacting with the bot
 
-You can interact with this bot by sending it a message with an image or a doodle. Be sure to add a message like "Turn this image into an Adaptive Card". As an example, you can use the image included in the `./assets` folder. Large resolution images will not work due to the limitations of the AI model.
+You can interact with this bot by sending it a message with an image or a doodle. Be sure to add a message like "Turn this image into an Adaptive Card". As an example, you can use the image included in the `./assets` folder. Large resolution images will not work due to the limitations of the AI model. Since the TeamsAttachmentDownloader is only using low resolution images, your image will be converted to 512px by 512px and budgeted at 85 tokens.
 
 ## Setting up the sample
 

js/samples/04.ai-apps/c.vision-cardGazer/src/index.ts

@@ -87,11 +87,11 @@ if (!process.env.OPENAI_KEY && !process.env.AZURE_OPENAI_KEY) {
 const model = new OpenAIModel({
     // OpenAI Support
     apiKey: process.env.OPENAI_KEY!,
-    defaultModel: 'gpt-4-vision-preview',
+    defaultModel: 'gpt-4o-mini',
 
     // Azure OpenAI Support
     azureApiKey: process.env.AZURE_OPENAI_KEY!,
-    azureDefaultDeployment: 'gpt-4-vision-preview',
+    azureDefaultDeployment: 'gpt-4o-mini',
     azureEndpoint: process.env.AZURE_OPENAI_ENDPOINT!,
     azureApiVersion: '2023-03-15-preview',
 
@@ -132,16 +132,29 @@ interface SendCardParams {
     card: any;
 }
 
-app.ai.action<SendCardParams>('SendCard', async (context, state, params) => {
+app.ai.action<SendCardParams>('SendAdaptiveCard', async (context, state, params) => {
     const attachment = CardFactory.adaptiveCard(params.card);
     await context.sendActivity(MessageFactory.attachment(attachment));
     return 'card sent';
 });
 
-app.ai.action<SendCardParams>('ShowCardJSON', async (context, state, params) => {
-    const json = JSON.stringify(params.card, null, 2);
-    await context.sendActivity(`<pre>${json}</pre>`);
-    return 'card displayed';
+app.ai.action<SendCardParams>('DisplayJSON', async (context, state, params) => {
+    const adaptiveCardJson = {
+        type: 'AdaptiveCard',
+        version: '1.6',
+        body: [
+            {
+                type: 'CodeBlock',
+                language: 'Json',
+                codeSnippet: JSON.stringify(params.card, null, 2)
+            }
+        ]
+    };
+
+    // Create the attachment
+    const attachment = CardFactory.adaptiveCard(adaptiveCardJson);
+    await context.sendActivity(MessageFactory.attachment(attachment));
+    return `json sent`;
 });
 
 // Listen for incoming server requests.

js/samples/04.ai-apps/c.vision-cardGazer/src/prompts/chat/actions.json

@@ -1,34 +1,32 @@
 [
     {
-        "name": "SendCard",
+        "name": "SendAdaptiveCard",
         "description": "Sends an adaptive card to the user",
         "parameters": {
             "type": "object",
             "properties": {
                 "card": {
+                    "$ref": "https://adaptivecards.io/schemas/adaptive-card.json#",
                     "type": "object",
                     "description": "The adaptive card to send"
                 }
             },
-            "required": [
-                "card"
-            ]
+            "required": ["card"]
         }
     },
     {
-        "name": "ShowCardJSON",
+        "name": "DisplayJSON",
         "description": "Shows the user the JSON for an adaptive card",
         "parameters": {
             "type": "object",
             "properties": {
                 "card": {
+                    "$ref": "https://adaptivecards.io/schemas/adaptive-card.json#",
                     "type": "object",
-                    "description": "The adaptive card JSON to show"
+                    "description": "The adaptive card JSON as raw text"
                 }
             },
-            "required": [
-                "card"
-            ]
+            "required": ["card"]
         }
     }
-]
+]

js/samples/04.ai-apps/c.vision-cardGazer/src/prompts/chat/config.json

@@ -3,7 +3,7 @@
     "description": "Vision Bot",
     "type": "completion",
     "completion": {
-        "model": "gpt-4-vision-preview",
+        "model": "gpt-4o-mini",
         "completion_type": "chat",
         "include_history": true,
         "include_input": true,

js/samples/04.ai-apps/c.vision-cardGazer/src/prompts/chat/skprompt.txt

@@ -1,4 +1,4 @@
 You are a friendly assistant for Microsoft Teams with vision support.
 You are an expert on converting doodles and images to Adaptive Cards for Microsoft Teams.
-When shown an image try to convert it to an Adaptive Card and send it using SendCard.
-For Adaptive Cards with Image placeholders use ShowCardJSON instead.
+When shown an image try to convert it to an Adaptive Card and send it using SendAdaptiveCard.
+When the user asks for JSON, use DisplayJSON to show the JSON of the Adaptive Card.

js/samples/04.ai-apps/c.vision-cardGazer/teamsapp.yml

@@ -113,3 +113,4 @@ publish:
     - uses: teamsApp/update
       with:
           appPackagePath: ./appPackage/build/appPackage.${{TEAMSFX_ENV}}.zip
+projectId: 18c89ad5-9038-4751-aee1-0a5143804bc3

js/samples/05.authentication/b.oauth-bot/README.md

@@ -2,7 +2,7 @@
 
 This sample shows how to incorporate a basic conversational SSO flow into a Microsoft Teams application using [Bot Framework](https://dev.botframework.com) and the Teams AI SDK.
 
-This sample requires creating an OAuth Connection in Azure Bot Service, which provides a token store to store the token after sign-in.
+This sample requires creating an OAuth Connection in Azure Bot Service, which provides a token store to store the token after sign-in. You may need to enable SSO for your app. See [Enable SSO for your app](https://learn.microsoft.com/en-us/microsoftteams/platform/bots/how-to/authentication/bot-sso-overview) for more information.
 
 Note that this bot will only work in tenants where the following graph scopes are permitted: