Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JS] feat: File download support, vision support, and vision sample #913

Closed
wants to merge 8 commits into from

Conversation

Stevenic
Copy link
Collaborator

@Stevenic Stevenic commented Nov 28, 2023

Linked issues

closes: #912
fixes: #786

Details

This change adds 2 key new features to the AI library:

  • A new file downloader feature lets developers reregister file downloader plugins that can download files relative to the users input. A TeamsAttachmentDownloader implementation is provided which simplifies the downloading of files uploaded to a bot.
  • GPT Vision support is added so that downloaded image files can be sent to GPT as part of the users input.

This PR also fixes a couple of prompt related bugs that were identified in the process.

Change details

  • Fixed bugs related to prompts not using the override options defined in a prompts config.json file.
  • Added a new InputFileDownloader plugin model to the Application class.
  • Added a TeamsAttachmentDownloader class for automatically downloading files uploaded to the bot.
  • Added a new state.temp.input_files variable which is of type InputFile[].
  • Moved the initialization of state.temp.input from the AI class to the Application class as it made more sense to do this earlier.
  • Updated the Message interface to support the new features needed for the vision API's.
  • Added a new UserInputMessage prompt section that knows how to format "user" messages containing images.
  • Updated ConversationHistory prompt section to count tokens for included images. This is currently hard coded to 85 which is the cost of low detail images. High detail images consume more tokens but it would require more work to count them properly.
  • Updated OpenAIModel class to return the LLMClient the input message it sent to the model. This is so the message with the images gets properly added to conversation history.

Attestation Checklist

  • My code follows the style guidelines of this project

  • I have checked for/fixed spelling, linting, and other errors

  • I have commented my code for clarity

  • I have made corresponding changes to the documentation (we use TypeDoc to document our code)

  • My changes generate no new warnings

  • I have added tests that validates my changes, and provides sufficient test coverage. I have tested with:

    • Local testing
    • E2E testing in Teams
  • New and existing unit tests pass locally with my changes

Additional information

Feel free to add other relevant information below

@Stevenic Stevenic requested review from a team, corinagum, aacebo and lilyydu as code owners November 28, 2023 21:40
Copy link
Collaborator

@corinagum corinagum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All TeamsChef references should be removed for non-Teams Chef bot samples

  • manifest images
  • screencaps
  • names

Copy link
Collaborator

@corinagum corinagum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All TeamsChef references should be removed for non-Teams Chef bot samples

  • manifest images
  • screencaps
  • names

corinagum
corinagum previously approved these changes Dec 6, 2023
@corinagum corinagum closed this Dec 6, 2023
corinagum added a commit that referenced this pull request Dec 6, 2023
…1018)

## Linked issues

closes: #912 
fixes: #786, #399

## NOTE 
> This PR is identical to
#913. The only difference is
the person creating the PR.

## Details

This change adds 2 key new features to the AI library:
- A new file downloader feature lets developers reregister file
downloader plugins that can download files relative to the users input.
A `TeamsAttachmentDownloader` implementation is provided which
simplifies the downloading of files uploaded to a bot.
- GPT Vision support is added so that downloaded image files can be sent
to GPT as part of the users input.

This PR also fixes a couple of prompt related bugs that were identified
in the process.

#### Change details

- Fixed bugs related to prompts not using the override options defined
in a prompts config.json file.
- Added a new `InputFileDownloader` plugin model to the `Application`
class.
- Added a `TeamsAttachmentDownloader` class for automatically
downloading files uploaded to the bot.
- Added a new `state.temp.input_files` variable which is of type
`InputFile[]`.
- Moved the initialization of `state.temp.input` from the `AI` class to
the `Application` class as it made more sense to do this earlier.
- Updated the `Message` interface to support the new features needed for
the vision API's.
- Added a new `UserInputMessage` prompt section that knows how to format
"user" messages containing images.
- Updated `ConversationHistory` prompt section to count tokens for
included images. This is currently hard coded to 85 which is the cost of
low detail images. High detail images consume more tokens but it would
require more work to count them properly.
- Updated `OpenAIModel` class to return the `LLMClient` the input
message it sent to the model. This is so the message with the images
gets properly added to conversation history.
 
## Attestation Checklist

- [x] My code follows the style guidelines of this project

- I have checked for/fixed spelling, linting, and other errors
- I have commented my code for clarity
- I have made corresponding changes to the documentation (we use
[TypeDoc](https://typedoc.org/) to document our code)
- My changes generate no new warnings
- I have added tests that validates my changes, and provides sufficient
test coverage. I have tested with:
  - Local testing
  - E2E testing in Teams
- New and existing unit tests pass locally with my changes

### Additional information

> Feel free to add other relevant information below

---------

Co-authored-by: Steven Ickman <[email protected]>
Co-authored-by: Corina Gum <>
Co-authored-by: Lily Du <[email protected]>
@aacebo aacebo mentioned this pull request Dec 21, 2023
@lilyydu lilyydu mentioned this pull request Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: add support for gpt-vision Azure OpenAI Deployment URL Endpoint is crafted wrongly
3 participants