VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) #8949

erikeldridge · 2025-04-18T21:48:53Z

#8943 enabled non-streaming methods to use Chrome's on-device model. This PR does the same for the streaming methods (generateContentStream and sendMessageStream).

Replaces #8918, since we're now merging into a non-main branch for prerelease.

We only define system prompts via onDeviceParams initialization.

changeset-bot · 2025-04-18T21:48:57Z

⚠️ No Changeset found

Latest commit: 7af0f8d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions · 2025-04-18T21:49:05Z

Vertex AI Mock Responses Check ⚠️

A newer major version of the mock responses for Vertex AI unit tests is available. update_vertexai_responses.sh should be updated to clone the latest version of the responses: v10.0

packages/util/src/environment.ts

packages/vertexai/package.json

packages/vertexai/src/methods/generate-content.ts

google-oss-bot · 2025-04-18T21:58:27Z

Size Report ¹

Affected Products

@firebase/vertexai
Type Base (e069751) Merge (60ce29e) Diff
browser 38.3 kB 39.2 kB +927 B (+2.4%)
main 39.3 kB 40.2 kB +927 B (+2.4%)
module 38.3 kB 39.2 kB +927 B (+2.4%)
firebase
Type Base (e069751) Merge (60ce29e) Diff
firebase-vertexai.js 30.8 kB 31.4 kB +654 B (+2.1%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/Uwk7zailSK.html

google-oss-bot · 2025-04-18T22:08:28Z

Size Analysis Report ¹

Affected Products

`@firebase/vertexai`

`ChatSession`

Size

Type	Base (`e069751`)	Merge (`60ce29e`)	Diff
size	16.1 kB	16.3 kB	+126 B (+0.8%)
size-with-ext-deps	35.1 kB	35.2 kB	+126 B (+0.4%)

Dependency

Type Base (e069751) Merge (60ce29e) Diff

functions

22 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
createEnhancedContentResponse
formatBlockErrorMessage
formatNewContent
generateContent
generateContentOnCloud
generateContentStream
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

23 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
createEnhancedContentResponse
formatBlockErrorMessage
formatNewContent
generateContent
generateContentOnCloud
generateContentStream
generateContentStreamOnCloud
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

+ generateContentStreamOnCloud

`GenerativeModel`

Size

Type	Base (`e069751`)	Merge (`60ce29e`)	Diff
size	19.5 kB	19.6 kB	+145 B (+0.7%)
size-with-ext-deps	38.5 kB	38.6 kB	+145 B (+0.4%)

Dependency

Type Base (e069751) Merge (60ce29e) Diff

functions

26 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

27 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateContentStreamOnCloud
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

+ generateContentStreamOnCloud

`ImagenModel`

Size

Type	Base (`e069751`)	Merge (`60ce29e`)	Diff
size	21.2 kB	21.4 kB	+145 B (+0.7%)
size-with-ext-deps	40.2 kB	40.4 kB	+145 B (+0.4%)

Dependency

Type Base (e069751) Merge (60ce29e) Diff

functions

28 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
createPredictRequestBody
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
handlePredictResponse
makeRequest
processStream
registerVertex
validateChatHistory

29 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
createPredictRequestBody
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateContentStreamOnCloud
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
handlePredictResponse
makeRequest
processStream
registerVertex
validateChatHistory

+ generateContentStreamOnCloud

`VertexAIModel`

Size

Type	Base (`e069751`)	Merge (`60ce29e`)	Diff
size	19.5 kB	19.6 kB	+145 B (+0.7%)
size-with-ext-deps	38.5 kB	38.6 kB	+145 B (+0.4%)

Dependency

Type Base (e069751) Merge (60ce29e) Diff

functions

26 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

27 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateContentStreamOnCloud
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

+ generateContentStreamOnCloud

`getGenerativeModel`

Size

Type	Base (`e069751`)	Merge (`60ce29e`)	Diff
size	21.6 kB	22.2 kB	+610 B (+2.8%)
size-with-ext-deps	40.6 kB	41.2 kB	+610 B (+1.5%)

Dependency

Type Base (e069751) Merge (60ce29e) Diff

functions

27 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateResponseSequence
getClientHeaders
getFunctionCalls
getGenerativeModel
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

28 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateContentStreamOnCloud
generateResponseSequence
getClientHeaders
getFunctionCalls
getGenerativeModel
getHeaders
getResponsePromise
getResponseStream
getText
hadBadFinishReason
makeRequest
processStream
registerVertex
validateChatHistory

+ generateContentStreamOnCloud

`getImagenModel`

Size

Type	Base (`e069751`)	Merge (`60ce29e`)	Diff
size	21.4 kB	21.6 kB	+145 B (+0.7%)
size-with-ext-deps	40.4 kB	40.5 kB	+145 B (+0.4%)

Dependency

Type Base (e069751) Merge (60ce29e) Diff

functions

29 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
createPredictRequestBody
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getImagenModel
getResponsePromise
getResponseStream
getText
hadBadFinishReason
handlePredictResponse
makeRequest
processStream
registerVertex
validateChatHistory

30 dependencies

addHelpers
aggregateResponses
assignRoleToPartsAndValidateSendMessageRequest
constructRequest
countTokens
countTokensOnCloud
createEnhancedContentResponse
createPredictRequestBody
formatBlockErrorMessage
formatGenerateContentInput
formatNewContent
formatSystemInstruction
generateContent
generateContentOnCloud
generateContentStream
generateContentStreamOnCloud
generateResponseSequence
getClientHeaders
getFunctionCalls
getHeaders
getImagenModel
getResponsePromise
getResponseStream
getText
hadBadFinishReason
handlePredictResponse
makeRequest
processStream
registerVertex
validateChatHistory

+ generateContentStreamOnCloud

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/t7CwuNHFFp.html

…tex-stream-rebased

…8949)

Fix languageCode parameter in action_code_url (#8912) * Fix languageCode parameter in action_code_url * Add changeset Vaihi add langmodel types. (#8927) * Adding LanguageModel types. These are based off https://github.com/webmachinelearning/prompt-api?tab=readme-ov-file#full-api-surface-in-web-idl * Adding LanguageModel types. * Remove bunch of exports * yarn formatted * after lint Define HybridParams (#8935) Co-authored-by: Erik Eldridge <[email protected]> Adding smoke test for new hybrid params (#8937) * Adding smoke test for new hybrid params * Use the existing name of the model params input --------- Co-authored-by: Erik Eldridge <[email protected]> Moving to in-cloud naming (#8938) Co-authored-by: Erik Eldridge <[email protected]> Moving to string type for the inference mode (#8941) Define ChromeAdapter class (#8942) Co-authored-by: Erik Eldridge <[email protected]> VinF Hybrid Inference: Implement ChromeAdapter (rebased) (#8943) Adding count token impl (#8950) VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) (#8949) Define values for Availability enum (#8951) VinF Hybrid Inference: narrow Chrome input type (#8953) Add image inference support (#8954) * Adding image based input for inference * adding image as input to create language model object disable count tokens api for on-device inference (#8962) VinF Hybrid Inference: throw if only_on_device and model is unavailable (#8965)

erikeldridge added 18 commits April 18, 2025 09:33

Define HybridParams

a3d869b

Copy over most types from @types package

14eee16

Trim unused AI types

b242749

Assert HybridParams sets the model name

5e97457

Use dom-chromium-ai package directly

1fe8a08

Define ChromeAdapter class

869fee7

Implement ChromeAdapter class

ff31b42

Integrate with e2e test app

1e487d5

Parameterize default in-cloud model name

8307fe5

Use type for inference mode and update docs

291c53b

Remove stray ai.ts

fe2bebc

Run yarn format

d4286d6

Test request-based availability checks

b898cd0

Remove request.systemInstruction validation

2fb2795

We only define system prompts via onDeviceParams initialization.

Integrate chrome adapter into stream methods

ef893c9

Refactor to emulate Vertex response

4c37859

Group response formatting methods together

eb25fec

Run docgen

b8d849c

erikeldridge requested a review from gsiddh April 18, 2025 21:48

erikeldridge requested a review from a team as a code owner April 18, 2025 21:48

erikeldridge mentioned this pull request Apr 18, 2025

VinF Hybrid Inference #4: ChromeAdapter in stream methods #8918

Closed

erikeldridge commented Apr 18, 2025

View reviewed changes

packages/util/src/environment.ts Outdated Show resolved Hide resolved

erikeldridge commented Apr 18, 2025

View reviewed changes

packages/vertexai/package.json Outdated Show resolved Hide resolved

erikeldridge commented Apr 18, 2025

View reviewed changes

packages/vertexai/src/methods/generate-content.ts Show resolved Hide resolved

Re-remove isChrome

1b9c98d

Re-remove dom-chromium-ai

5092bd8

erikeldridge added 3 commits April 18, 2025 16:51

Unit test stream method

025b786

Remove redundant ondevice suffix

34c658e

Merge remote-tracking branch 'public/vaihi-exp' into erikeldridge-ver…

7af0f8d

…tex-stream-rebased

erikeldridge merged commit 3d7df8b into vaihi-exp Apr 19, 2025
32 of 45 checks passed

erikeldridge deleted the erikeldridge-vertex-stream-rebased branch April 19, 2025 00:53

gsiddh pushed a commit that referenced this pull request Apr 22, 2025

VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) (#…

23a8c86

…8949)

gsiddh pushed a commit that referenced this pull request Apr 22, 2025

VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) (#…

58d6d1f

…8949)

gsiddh mentioned this pull request Apr 23, 2025

Complete Hybrid inference impl #8966

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) #8949

VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) #8949

erikeldridge commented Apr 18, 2025

changeset-bot bot commented Apr 18, 2025 •

edited

Loading

github-actions bot commented Apr 18, 2025

google-oss-bot commented Apr 18, 2025 •

edited

Loading

`@firebase/vertexai`

`firebase`

google-oss-bot commented Apr 18, 2025 •

edited

Loading

`@firebase/vertexai`

`ChatSession`

`GenerativeModel`

`ImagenModel`

`VertexAIModel`

`getGenerativeModel`

`getImagenModel`

VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) #8949

VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) #8949

Conversation

erikeldridge commented Apr 18, 2025

changeset-bot bot commented Apr 18, 2025 • edited Loading

⚠️ No Changeset found

github-actions bot commented Apr 18, 2025

Vertex AI Mock Responses Check ⚠️

google-oss-bot commented Apr 18, 2025 • edited Loading

Size Report 1

Affected Products

@firebase/vertexai

firebase

Test Logs

google-oss-bot commented Apr 18, 2025 • edited Loading

Size Analysis Report 1

Affected Products

@firebase/vertexai

ChatSession

GenerativeModel

ImagenModel

VertexAIModel

getGenerativeModel

getImagenModel

Test Logs

changeset-bot bot commented Apr 18, 2025 •

edited

Loading

google-oss-bot commented Apr 18, 2025 •

edited

Loading

Size Report ¹

`@firebase/vertexai`

`firebase`

google-oss-bot commented Apr 18, 2025 •

edited

Loading

Size Analysis Report ¹

`@firebase/vertexai`

`ChatSession`

`GenerativeModel`

`ImagenModel`

`VertexAIModel`

`getGenerativeModel`

`getImagenModel`