Use Prime CLI config for v1 eval#1703
Conversation
ApprovabilityVerdict: Needs human review Changes authentication credential resolution logic with potential to send Prime API keys to unintended endpoints. Two unresolved review comments identify security concerns around credential handling that warrant human review. You can customize Macroscope's approvability policy. Learn more. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04b33e8247
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not api_key and config.api_key_var == "PRIME_API_KEY": | ||
| api_key = load_prime_config().get("api_key") |
There was a problem hiding this comment.
When a user sets only --client.base-url to a non-Prime endpoint while logged in to Prime, this branch still pulls the Prime API key from ~/.prime/config.json because api_key_var remains the default. That sends a live Prime bearer token to whichever custom URL was explicitly configured; previously these runs used EMPTY unless PRIME_API_KEY was set in the environment. Gate the Prime CLI fallback on the default/Prime base URL, or require an explicit opt-in, so custom endpoints do not receive Prime credentials.
Useful? React with 👍 / 👎.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e1d8226. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1d8226ce4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if config.base_url.rstrip("/") == prime_base_url.rstrip("/"): | ||
| api_key = prime_config.get("api_key") |
There was a problem hiding this comment.
When a user overrides the endpoint back to the built-in Prime URL while their active PRIME_INFERENCE_URL or CLI inference_url points somewhere else, this exact-match gate prevents the Prime CLI api_key fallback and the client sends EMPTY instead. For example, a logged-in user with PRIME_INFERENCE_URL set to a staging router who runs one eval with --client.base-url https://api.pinference.ai/api/v1 will get auth failures unless they also export PRIME_API_KEY, even though the change is meant to preserve explicit endpoint overrides while still using Prime login credentials.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ff7fe1c3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if "base_url" not in self.model_fields_set: | ||
| self.base_url = prime_base_url |
There was a problem hiding this comment.
model_fields_set only contains values supplied by the caller, not defaults declared on a BaseClientConfig subclass. With an environment config such as class StagingJudge(BaseClientConfig): base_url = "https://staging.pinference.ai/api/v1" that intentionally keeps api_key_var="PRIME_API_KEY", this branch treats the subclass default as missing and overwrites it with the active Prime/default URL, so the environment silently calls the wrong endpoint unless every caller explicitly passes base_url.
Useful? React with 👍 / 👎.
| host = urlparse(config.base_url).hostname or "" | ||
| if ( | ||
| not api_key | ||
| and config.api_key_var == "PRIME_API_KEY" | ||
| and (host == PRIME_INFERENCE_HOST or host.endswith(f".{PRIME_INFERENCE_HOST}")) | ||
| ): | ||
| api_key = load_prime_config().get("api_key") |
There was a problem hiding this comment.
arfh, i hate this custom logic here. getting auth smell. how come no other provider requires this?
2 participants
Overview
Make direct v1
uv run evalcalls use the active Prime CLI inference settings when the default Prime client is selected.Details
~/.prime/config.jsonfor the API key and inference URL when matching environment variables are absent.Note
Medium Risk
Changes credential resolution and team billing headers for Prime inference; wrong fallbacks could mis-route billing or send requests without a valid key.
Overview
Direct
uv run evalwith the default Prime client now picks up the same inference URL, API key, and team billing asprime eval, by falling back to~/.prime/config.jsonwhen env vars are missing.BaseClientConfigreplacesadd_prime_team_idwithapply_prime_config: only whenapi_key_varisPRIME_API_KEY, it resolvesbase_urlfromPRIME_INFERENCE_URL→ CLIinference_url→ default, and setsX-Prime-Team-IDfromPRIME_TEAM_IDor CLIteam_idonly if the host is Prime inference. Explicitbase_urlin config still wins.resolve_clientloads the API key from the Prime CLI config whenPRIME_API_KEYis unset and the endpoint is Prime-hosted, instead of defaulting straight to"EMPTY".Reviewed by Cursor Bugbot for commit 4ff7fe1. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Use Prime CLI config as fallback for base URL, team ID, and API key in v1 eval
BaseClientConfig.apply_prime_config(in config.py) now setsbase_urlfromPRIME_INFERENCE_URLenv var or CLI config when not explicitly provided, and injects theX-Prime-Team-IDheader fromPRIME_TEAM_IDenv var or CLI config forpinference.aihosts.resolve_clientfalls back to the Prime CLI configapi_keyforpinference.aihosts when the expected env var is unset, reducing cases where'EMPTY'is used as the key.pinference.aithat previously required explicit env vars will now silently pick up values from the local Prime CLI config if those vars are absent.Macroscope summarized 4ff7fe1.