Ability to set `maxTokens` when calling `prompt()`

[duci9y]

I'd like to be able to constrain the output to N tokens for my use case (tab autocompletion). Otherwise the model generates strings that are too long to be useful. It also takes more time.

[etiennenoel]

Do you have a use case for N tokens or number of words? I am tempted to think that developers will probably want a number of words and if that's the case,

@domenic Should we ask devs to do this using Regex with Structured Output?

[domenic]

Although I agree there is some overlap with response constraints, I think there's sufficient precedent in server-side APIs for exactly this constraint such that we might want to include it. Additionally, we already expose token limits in the API (via inputUsage / inputQuota).

It's possible most people using this on server-side APIs are concerned about costs, which are not as applicable for built-in AI. But here we have at least one user suggesting it's about usefulness and time taken.

What I can't easily find out from public docs is how this is implemented in existing server-side APIs. Does it get piped to the model, so that the model tries to stay within the limit? Or is it just a hard cutoff on the frontend, whenever the limit is reached?

Some experimentation in the OpenAI playground implies it's the latter: