Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ one.
| Multi-need and structured MCP | Sequential and steer delivery, bounded ledger, structured JSON tools, cancellation, shared resolver | App Server simulator and one structured MCP cache-hit observation | **Implemented; offline validated; live calibration** |
| Claim-level reuse | Validator-extracted claims, claim proofs, mixed planning, and bounded authoritative location, runtime-flow, and focused-test claims | Deterministic freshness, mutation, negative, projection, economics, and performance cases | **Implemented; offline validated** |
| Verified changes | Isolated patch preparation, independent verifier, one repair, explicit journaled apply | Simulator and focused persistence, isolation, drift, and recovery tests | **Implemented; offline validated** |
| Codex role-profile control plane | Canonical Codex role definitions, bounded policies, immutable revisions, state-digest CAS, SQLite V14 persistence, audit records, explicit WorkerProfile projection, bounded digest-bound HTTP API, and local editor | Focused deterministic Rust and frontend tests; configuration-only boundary (no worker/session binding or lifecycle execution) | **Implemented; offline validated** |
| Codex role-profile control plane | Canonical Codex role definitions, bounded policies, immutable revisions, state-digest CAS, SQLite persistence, audit records, explicit WorkerProfile projection, bounded digest-bound HTTP API, local editor, and frozen session/worker/cache provenance | Focused deterministic Rust and frontend tests; no parent-owned lifecycle execution | **Implemented; offline validated** |
| Codex development lifecycle orchestration | Evidence, patch, test, verification, approval, and apply primitives exist; the configurable parent-owned role lifecycle is not integrated | Component-level offline evidence only | **Pending** |
| Other-host subagent configuration | Configuration-only interoperability is planned for Claude Code and Cursor, followed by OpenCode and Antigravity | Not available | **Pending** |
| Multi-host orchestration | Execution remains Codex-only; non-Codex execution follows configuration interoperability, a host contract, and conformance evidence | Not available | **Pending** |
Expand Down Expand Up @@ -61,8 +61,9 @@ provider-backed claim-authority observation exists.
| Embedded React control plane | **Implemented; frontend and local end-to-end validation** |
| Needs, proofs, claims, changes, runs, models, cache, settings, approvals | **Implemented; development interface** |
| Canonical named Codex role-profile domain and revision store | **Implemented; offline validated** |
| Named role-profile HTTP/editor | **Implemented; offline validated; configuration-only** |
| Role-profile session binding and lifecycle integration | **Pending; Codex-first** |
| Named role-profile HTTP/editor | **Implemented; offline validated; configuration mutations only** |
| Role-profile session, worker, cache, attempt, and audit provenance | **Implemented; offline validated; Codex-first** |
| Parent-owned role-profile lifecycle integration | **Pending; Codex-first** |
| Non-Codex subagent configuration | **Pending; configuration only before execution** |
| Non-Codex execution and orchestration | **Pending; later milestone** |
| Stable public API or configuration compatibility | **Pending** |
Expand Down Expand Up @@ -125,9 +126,10 @@ validation.
provider-backed evidence.
- Verified changes have no provider-backed patcher or verifier observation.
- Canonical role-profile definitions, revision persistence, bounded HTTP/editor
flows, and request-time preflight are implemented and offline validated.
They do not provide session or worker binding, a lifecycle executor, or
automatic profile activation; activation is an explicit configuration change.
flows, request-time preflight, and frozen session/worker/cache/attempt/audit
provenance are implemented and offline validated. They do not provide a
parent-owned lifecycle executor or automatic profile activation; activation
is an explicit configuration change.
- The verifier handles a deterministic serial set of up to four distinct
associated certified test plans; exact duplicates collapse to one execution,
while over-cap and unavailable plans fail closed. This behavior is offline
Expand Down
17 changes: 15 additions & 2 deletions crates/needle-app/src/artifact_cache_main_replay.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use super::{
AppError, HookConfig, absolute_run_path, canonical_child_path, option_value,
repository_status_clean, required_value, resolve_codex,
provision_experiment_role_profile, repository_status_clean, required_value, resolve_codex,
};
use needle_bench::{ArtifactCacheReplayReport, run_artifact_cache_replay};
use needle_core::{
Expand Down Expand Up @@ -47,6 +47,7 @@ impl WorkerExecutor for ForbiddenWorker {
discarded_facts: 0,
worker_session_id: None,
session_cleanup_success: Some(true),
role_profile_provenance: None,
}))
}
}
Expand Down Expand Up @@ -102,6 +103,16 @@ pub(super) fn run(arguments: &[String]) -> Result<(), AppError> {
let store = RuntimeStore::new(artifact_root.join("needle.sqlite3"));
let profile =
HookConfig::default().profile().map_err(|error| AppError::Experiment(error.to_string()))?;
let role_profile_id = provision_experiment_role_profile(
&store,
"artifact-cache-replay.explorer",
profile.definition_digest,
"recorded-r35-fixture",
"low",
"default",
1,
false,
)?;
let instructions = profile.rendered_context_owned();
let main_config = WorkerConfig {
executable: simulator.display().to_string(),
Expand All @@ -110,6 +121,7 @@ pub(super) fn run(arguments: &[String]) -> Result<(), AppError> {
service_tier: Some("default".to_owned()),
timeout_seconds: 10,
evidence_failure_policy: EvidenceFailurePolicy::DiscardInvalidFact,
role_profile_provenance: None,
};
let mut session = CodexMainSession::start(MainSessionConfig {
codex: &main_config,
Expand All @@ -126,11 +138,12 @@ pub(super) fn run(arguments: &[String]) -> Result<(), AppError> {
.map_err(AppError::Experiment)?;
let session_id = session.thread_id().to_owned();
store
.record_session_start(
.record_session_start_profiled(
&session_id,
profile.definition_digest,
Some("simulated-main-r35-cache"),
source_repository.to_str(),
&role_profile_id,
)
.map_err(|error| AppError::Experiment(error.to_string()))?;

Expand Down
122 changes: 110 additions & 12 deletions crates/needle-app/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ use needle_bench::{
evaluate_pilot_pair, parse_codex_jsonl, parse_jsonl, parse_task_fixture, redact_jsonl,
};
use needle_core::{
Digest, EvidenceFailurePolicy, FORMAT_REVISION, NeedKey, NeedRequest, TestPlan, WorkerConfig,
CodexHost, CodexRole, CommandPolicy, Digest, EvidenceFailurePolicy, FORMAT_REVISION,
FallbackPolicy, FilesystemPolicy, NeedKey, NeedRequest, NetworkPolicy, ReasoningLevel,
RepairPolicy, RoleProfileBudget, RoleProfileDefinition, RoleProfileDefinitionInput,
RoleProfileId, ServiceTier, TestPlan, TestPolicy, ToolPolicy, WorkerConfig,
};
use needle_platform_codex::{
CodexWorker, CompactInput, HookConfig, SessionEndInput, SessionStartInput, StopInput,
Expand Down Expand Up @@ -508,15 +511,33 @@ fn run_hook(arguments: Vec<String>) -> Result<(), AppError> {
if let Some(session_id) = parsed.session_id.as_deref() {
let store = hook_runtime_store()?;
let profile_digest = config.profile()?.definition_digest;
if let Err(error) = store.initialize().and_then(|_| {
store.record_session_start(
session_id,
profile_digest,
parsed.model.as_deref(),
parsed.cwd.as_deref(),
)
}) {
eprintln!("needle: cannot record product session ({error}); fail-open");
let selector = env::var("NEEDLE_ROLE_PROFILE_ID").ok();
let result = match selector {
Some(value) => match RoleProfileId::new(value) {
Ok(profile_id) => store.initialize().and_then(|_| {
store.record_session_start_profiled(
session_id,
profile_digest,
parsed.model.as_deref(),
parsed.cwd.as_deref(),
&profile_id,
)
}),
Err(error) => Err(needle_runtime::StoreError::RoleProfileValidation(
error.to_string(),
)),
},
None => {
eprintln!(
"needle: NEEDLE_ROLE_PROFILE_ID is missing; session provenance is unknown"
);
Ok(())
}
};
if let Err(error) = result {
eprintln!(
"needle: cannot record profiled product session ({error}); fail-open"
);
}
}
serde_json::to_value(output)?
Expand Down Expand Up @@ -675,19 +696,24 @@ fn run_mcp(arguments: Vec<String>) -> Result<(), AppError> {
.or_else(|| env::var("NEEDLE_MCP_MAIN_MODEL").ok())
.unwrap_or_else(|| "unknown".to_owned());
validate_model_value(&main_model, "main model")?;
let role_profile = required_value(&arguments, "--role-profile").and_then(|value| {
RoleProfileId::new(value)
.map_err(|error| AppError::Usage(format!("invalid role profile: {error}")))
})?;
return mcp::serve(mcp::ProductMcpConfig {
data_directory,
repository_root,
main_model,
cache_only: arguments.iter().any(|argument| argument == "--cache-only"),
calibration_reuse: env::var("NEEDLE_INTERNAL_CALIBRATION_REUSE").as_deref()
== Ok("partial-tests-live"),
role_profile_id: role_profile,
})
.map_err(AppError::Runtime);
}
if arguments.first().map(String::as_str) != Some("serve-benchmark") || arguments.len() != 1 {
return Err(AppError::Usage(
"mcp serve [--data-dir <directory>] [--repository <root>] [--main-model <model>] [--cache-only] | mcp serve-benchmark"
"mcp serve --role-profile <id> [--data-dir <directory>] [--repository <root>] [--main-model <model>] [--cache-only] | mcp serve-benchmark"
.to_owned(),
));
}
Expand All @@ -699,7 +725,7 @@ fn validate_mcp_serve_arguments(arguments: &[String]) -> Result<(), AppError> {
while index < arguments.len() {
match arguments[index].as_str() {
"--cache-only" => index += 1,
"--data-dir" | "--repository" | "--main-model" => {
"--data-dir" | "--repository" | "--main-model" | "--role-profile" => {
if arguments.get(index + 1).is_none() {
return Err(AppError::Usage(format!("{} requires a value", arguments[index])));
}
Expand Down Expand Up @@ -808,6 +834,7 @@ fn transport_preflight_run(arguments: &[String]) -> Result<(), AppError> {
service_tier: Some(service_tier),
timeout_seconds: 30,
evidence_failure_policy: EvidenceFailurePolicy::DiscardInvalidFact,
role_profile_provenance: None,
};
let report = CodexWorker::new(&data_root)
.preflight_transport(&config, &repository)
Expand Down Expand Up @@ -3058,6 +3085,77 @@ fn validate_model_value(value: &str, label: &str) -> Result<(), AppError> {
validate_slug(value, label)
}

#[allow(clippy::too_many_arguments)]
fn provision_experiment_role_profile(
store: &RuntimeStore,
profile_id: &str,
prompt_profile_digest: Digest,
model: &str,
reasoning: &str,
service_tier: &str,
timeout_seconds: u64,
repair_once: bool,
) -> Result<RoleProfileId, AppError> {
let reasoning = match reasoning {
"low" => ReasoningLevel::Low,
"medium" => ReasoningLevel::Medium,
"high" => ReasoningLevel::High,
"xhigh" => ReasoningLevel::Xhigh,
value => {
return Err(AppError::Experiment(format!(
"experiment role profile cannot represent reasoning `{value}`"
)));
}
};
let service_tier = match service_tier {
"default" => ServiceTier::Default,
"priority" => ServiceTier::Priority,
value => {
return Err(AppError::Experiment(format!(
"experiment role profile cannot represent service tier `{value}`"
)));
}
};
let profile_id =
RoleProfileId::new(profile_id).map_err(|error| AppError::Experiment(error.to_string()))?;
let definition = RoleProfileDefinition::new(RoleProfileDefinitionInput {
profile_id: profile_id.clone(),
role: CodexRole::Explorer,
host: CodexHost::Codex,
model: model.to_owned(),
reasoning,
service_tier,
timeout_seconds,
budget: RoleProfileBudget {
max_turns: 8,
max_output_tokens: 2000,
max_cost_microusd: 1_000_000_000,
},
prompt_profile_digest,
output_contract_digest: Digest::blake3(needle_core::ARTIFACT_RESULT_SCHEMA_ID),
tool_policy: ToolPolicy::ReadOnly,
command_policy: CommandPolicy::ReadOnly,
filesystem_policy: FilesystemPolicy::ReadOnlyCheckout,
network_policy: NetworkPolicy::Denied,
test_policy: TestPolicy::Disabled,
repair_policy: if repair_once { RepairPolicy::Once } else { RepairPolicy::None },
fallback_policy: FallbackPolicy::Disabled,
concurrency: 1,
route_assignments: Vec::new(),
})
.map_err(|error| AppError::Experiment(error.to_string()))?;
let revision = store
.create_role_profile(definition)
.map_err(|error| AppError::Experiment(error.to_string()))?;
let state = store
.role_profile_state(&profile_id)
.map_err(|error| AppError::Experiment(error.to_string()))?;
store
.activate_role_profile(&profile_id, revision.revision, state.state_digest)
.map_err(|error| AppError::Experiment(error.to_string()))?;
Ok(profile_id)
}

fn parse_experiment_arm(value: &str) -> Result<ExperimentArm, AppError> {
match value {
"P0" => Ok(ExperimentArm::P0),
Expand Down
Loading