Updates ai-revision workflow for upcoming manubot-ai-editor multi-provider feature #522

falquaddoomi · 2025-02-28T18:57:45Z

We'll shortly be introducing the ability for manubot-ai-editor to work with multiple providers (e.g., OpenAI, Anthropic, and more), facilitated by the LangChain library. These changes are included in PR manubot/manubot-ai-editor#71. That PR should be merged and a release created before we merge this PR, so this PR will remain a draft until that's completed.

Since we'll potentially be implementing many providers, we decided to allow a generic environment variable, PROVIDER_API_KEY, to specify the API key used for any one provider. While manubot-ai-editor could theoretically be run with multiple providers (each of which would need their own valid API key, of course), this particular workflow always runs the revision process with a single specific provider, so I assumed that using PROVIDER_API_KEY would be sufficient here. To not break things during the transition, the value will also be retrieved from secrets.OPENAI_API_KEY if secrets.PROVIDER_API_KEY is unavailable; this will likely be removed down the road.

I've also added model_provider to the inputs to specify the model provider (i.e., "openai" or "anthropic" for now). I'm unsure if we should provide the list of options as a dropdown and just update this whenever we add new providers, or keep it as a free-text string. I've included the list of providers we currently support for now, but I'm considering removing that in favor of either having a set list of options that we update, or directing the user to the manubot-ai-editor docs for an up-to-date list.

Finally, on @miltondp's advice, the default model engine is now gpt-4o rather than gpt-4-turbo.

…I key if available

agitter

These generalization changes make sense and will be fast to approve. Let me know when you are ready to take it out of draft.

agitter · 2025-02-28T19:53:26Z

.github/workflows/ai-revision.yaml

+          commit-message: 'revise using AI model\n\nUsing the ${{ inputs.model_provider }} model ${{ inputs.model }}'
          title: 'AI-based revision using ${{ inputs.model }}'
-          author: OpenAI model ${{ inputs.model }} <support@openai.com>
+          author: ${{ inputs.model_provider }} model ${{ inputs.model }} <support@${{ inputs.model_provider }}.com>


Will that support email address always work for future providers?

Hey @agitter, thanks for the comment. Frankly, no, it won't always work; it's just chance that both OpenAI and Anthropic have support emails at a domain that can be easily constructed from the provider string. I also don't know how helpful these providers' support emails would be anyway.

I could see a few ways forward here:

change it to an email address for the Manubot AI Editor maintainers (my preferred option, since we'd be in a position to help if someone did have a comment)

add a workflow parameter that specifies the author email address (not my favorite option, since it's one more thing for the user to have to worry about)

use <${{ github.actor_id }}+${{ github.actor }}@users.noreply.github.com> (the default for the create-pull-request action) which would associate the commit with the user who ran the workflow

I'll talk to @miltondp, the owner of Manubot AI Editor, about the above options and revise the PR accordingly.

Thank you both for your work and comments. I'd go either with the first/preferred option or the third one. Or even not specify any email address if possible.

The first option seems less suitable to me because the Manubot AI Editor maintainers are the authors of the commit. Those maintainers shouldn't appear as contributing directly to many downstream manubot manuscripts. The third option is a reasonable default.

Thanks for the input, and that makes sense to me.

I've implemented option 3 as of commit 353d845. The commit author email is set to the same thing as the default value for the author field from the create-pull-request action inputs.

The display name remains <model_provider> model <model_name>; a example full author string would be:
openai model gpt-4-turbo <[email protected]>.

I've verified that the GitHub user who submitted the workflow is listed as a co-author on the commit; you can see an example of that in my fork of rootstock here: falquaddoomi@b8f36b0.

…PR comments

… patched manubot-ai-editor over it.

…ich uses the provider's default model.

…uputs it

…ulate PR details

… 'openai' with a warning

falquaddoomi · 2025-04-23T19:18:56Z

Summary of recent changes:

In order to cut down on the number of options a user needs to select when dispatching the workflow, a few options can now be specified as org/repo variables, specifically:
- AI_EDITOR_MODEL_PROVIDER: e.g., "openai" or "anthropic"; if "(repo default)" is selected as the model provider, this variable will be used to identify the model provider
- AI_EDITOR_LANGUAGE_MODEL: the model from the specified provider to use; the user's choice always overrides this, but if they specify a falsy value (e.g, a blank string), it'll fall back to using this variable. If the variable isn't available, either, the default language model as defined in manubot-ai-editor for the provider will be used.
The actual provider and model used are now scraped from manubot-ai-editor's output. I made this change mainly because the tool is the authoritative source on what is actually being used, but it's also is necessary when we defer the choice of the language model to manubot-ai-editor.

miltondp · 2025-04-24T15:00:42Z

.github/workflows/ai-revision.yaml

        required: true
        type: string
-        default: 'ai-revision-gpt4turbo'
+        default: 'ai-revision-results'


This is not high priority, but is it possible to use a variable here so that if we revise multiple times using a different provider/model, the output branch will not override the (potentially) existing one?

Again, not high priority.

I'm not sure, but that'd be a good idea; I'll look into it, thanks!

This is addressed in cc47cdd. The branch name field now defaults to ai-revision-{1}, where {1} is replaced with the final model name to match the older behavior. The provider could also be added to the branch name by specifying {0}.

As of 6b08f90, the new branch defaults to ai-revision-{0}-{1}, where {0} is the provider ("openai") and {1} is the model. I figured having more information in the branch name is better, especially since both values are short.

.github/workflows/ai-revision.yaml

… support in the same repo

Added early-exit flags so that workflow stops if manubot-ai-editor fails

Adds provider to default branch name

Adds body to PR with some details about the revision and instructions.

…stalls latest manubot-ai-editor >= 0.5.0

falquaddoomi · 2025-06-06T18:02:25Z

Hi @agitter; I believe that this PR is ready for merging now. (@miltondp, mentioning you for visibility.)

Feel free to review it again if you like, since there have been some changes since the last time. I've tested it pretty extensively in my fork, but more testing can't hurt, I suppose. Thanks!

miltondp

I think this looks good, @falquaddoomi. I left a minor question/suggestion, if it makes sense. I would wait for @agitter for the final review and merge.

miltondp · 2025-06-06T18:55:39Z

.github/workflows/ai-revision.yaml

        env:
-          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-          AI_EDITOR_LANGUAGE_MODEL: ${{ inputs.model }}
+          PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY || secrets.OPENAI_API_KEY }}


Should this also include Anthropic's, like in the suggestion below?

Suggested change

PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY || secrets.OPENAI_API_KEY }}

PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY || secrets.OPENAI_API_KEY || secrets.ANTHROPIC_API_KEY }}

Hey, thanks for the comment. I'd originally added that to not break compatibility with users who hadn't set the new PROVIDER_API_KEY variable, but still had OPENAI_API_KEY set.

Now that I think about it, though, it's not necessary for it to check OPENAI_API_KEY, since OPENAI_API_KEY will take precedence over PROVIDER_API_KEY for OpenAI models, and if the user chooses an Anthropic model it'll fail earlier with a missing key rather than trying and failing to use an OpenAI PROVIDER_API_KEY with Anthropic later on in th eprocess.

I'll change it to just use secrets.PROVIDER_API_KEY; thanks again!

Co-authored-by: Milton Pividori <[email protected]>

falquaddoomi added 2 commits February 26, 2025 13:06

Updates ai-revision workflow to select model provider, use generic AP…

ff9376d

…I key if available

Infers support email from provider. Default engine is now gpt-4o.

69e10e8

falquaddoomi mentioned this pull request Feb 28, 2025

Generalizes GPT3CompletionModel to work with other providers, adds Anthropic manubot/manubot-ai-editor#71

Merged

agitter reviewed Feb 28, 2025

View reviewed changes

falquaddoomi added 2 commits March 3, 2025 13:16

Updated commit author email to be the user running the workflow, per …

353d845

…PR comments

Fixes newlines in commit message not showing up correctly

6114452

agitter approved these changes Mar 14, 2025

View reviewed changes

falquaddoomi added 8 commits April 8, 2025 15:50

ai-revision workflow now uses langchain-anthropic-integration head

7cfbcf7

Fixed pkg installed; first installs manubot[ai-rev] and then installs…

2c7185a

… patched manubot-ai-editor over it.

Changed commit to branch so we don't have to update as often

dc514f9

Allows provider to be set via repo vars. Allows model to be blank, wh…

95864d2

…ich uses the provider's default model.

Removed default value for model input

2fcf741

Removed output of provider, model from workflow, since the tool now o…

f22809b

…uputs it

Scrapes resolved provider, model from manubot ai editor output to pop…

9f015d8

…ulate PR details

If provider (repo default) selected and no default is set, resorts to…

cd0613c

… 'openai' with a warning

falquaddoomi requested a review from miltondp April 23, 2025 19:10

miltondp reviewed Apr 24, 2025

View reviewed changes

.github/workflows/ai-revision.yaml Outdated Show resolved Hide resolved

miltondp reviewed Apr 24, 2025

View reviewed changes

.github/workflows/ai-revision.yaml Show resolved Hide resolved

falquaddoomi added 2 commits April 24, 2025 11:31

Re-adds model name, allows provider name in branch_name

cc47cdd

Adds optional provider-specific API key secrets for mulitple provider…

eafdc6c

… support in the same repo

falquaddoomi mentioned this pull request May 1, 2025

Adds provider, model vars docs to README manubot/manubot-ai-editor#88

Merged

falquaddoomi added 3 commits May 7, 2025 14:03

Update ai-revision.yaml

21eeed1

Added early-exit flags so that workflow stops if manubot-ai-editor fails

Update ai-revision.yaml

6b08f90

Adds provider to default branch name

Update ai-revision.yaml

0c3ca7b

Adds body to PR with some details about the revision and instructions.

falquaddoomi mentioned this pull request May 14, 2025

Write structured metadata after Manubot AI Editor run manubot/manubot-ai-editor#92

Open

falquaddoomi added 2 commits May 21, 2025 13:18

Testing manubot-ai-editor main branch instead of merged PR branch

b79222f

Reverts to installing manubot-ai-editor via manubot[ai-rev], which in…

ccbc45f

…stalls latest manubot-ai-editor >= 0.5.0

falquaddoomi marked this pull request as ready for review May 21, 2025 22:02

falquaddoomi marked this pull request as draft May 21, 2025 22:16

falquaddoomi marked this pull request as ready for review June 4, 2025 18:58

miltondp approved these changes Jun 12, 2025

View reviewed changes

PROVIDER_API_KEY no longer gets set to OPENAI_API_KEY as a fallback.

9b2feb8

Co-authored-by: Milton Pividori <[email protected]>

	PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY \|\| secrets.OPENAI_API_KEY }}
	PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY \|\| secrets.OPENAI_API_KEY \|\| secrets.ANTHROPIC_API_KEY }}

Updates ai-revision workflow for upcoming manubot-ai-editor multi-provider feature #522

Are you sure you want to change the base?

Updates ai-revision workflow for upcoming manubot-ai-editor multi-provider feature #522

Uh oh!

Conversation

falquaddoomi commented Feb 28, 2025

Uh oh!

agitter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

falquaddoomi Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

falquaddoomi commented Apr 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

falquaddoomi May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

falquaddoomi commented Jun 6, 2025

Uh oh!

miltondp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

falquaddoomi Feb 28, 2025 •

edited

Loading

falquaddoomi May 21, 2025 •

edited

Loading