[Future] Evaluate Just-in-Time Context Loading Strategy

[frankbria]

Summary

Evaluate implementing Just-in-Time (JIT) context loading to fetch relevant context only when immediately needed, rather than pre-loading based on tier classification.

Background: State of the Art

Philipp Schmid's context engineering framework emphasizes "just-in-time" loading as a key optimization. From his practical tips:

"Instead of pre-loading all data (traditional RAG), use just-in-time strategies."

The core insight: pre-loading context based on predicted relevance introduces two problems:

1. Latency penalty - loading context that may never be used
2. Relevance decay - context loaded early may be stale by the time it's needed

JIT loading means the agent requests specific context at the moment a tool call or decision requires it, ensuring maximum relevance and minimum waste.

Current State in CodeFRAME

The tiered HOT/WARM/COLD memory system assigns importance scores and manages context retention. However, it's unclear whether:

• Context is loaded proactively based on tier (pre-loading)
• Context is fetched on-demand when the LLM signals need (JIT)
• There's a hybrid approach

The flash_save mechanism handles persistence, but the loading strategy needs examination.

Investigation Tasks

1. Audit current loading behavior

   • Trace when context moves from COLD → WARM → HOT
   • Identify if loading is triggered by tier promotion rules or by explicit LLM requests
   • Measure how often loaded context is actually used in subsequent calls

2. Benchmark current approach

   • Track "context utilization rate" (loaded tokens vs. tokens referenced in responses)
   • Measure time-to-first-token with current loading strategy

3. Design JIT loading mechanism

   • Define tool/function for agents to request specific context categories
   • Implement lazy loading from COLD storage
   • Consider prefetch hints based on task type (e.g., "test task" prefetches test history)

4. Prototype and compare

   • A/B test pre-load vs. JIT for representative tasks
   • Measure token efficiency, latency, and task success rate

Success Criteria

• Documented understanding of current loading behavior
• Metrics showing context utilization rate
• If JIT shows benefit: implementation with measurable improvement in token efficiency
• If pre-loading is optimal: documented rationale for current approach

References

• Context Engineering - Philipp Schmid
• 12-Factor Agents - Own Your Context Window
• Manus data point: ~50 tool calls per task, context management critical

[traycerai]

Plan

Observations

The current CodeFRAME implementation uses a pre-loading strategy where both HOT and WARM context tiers are loaded before task execution (file:codeframe/agents/hybrid_worker.py lines 153-154). However, comments in file:codeframe/lib/importance_scorer.py line 160 indicate WARM tier was designed for "on-demand loading." The system loads all WARM items but only uses the first 20 in prompts (line 226), suggesting significant waste. There's no mechanism to track context utilization rates or enable agents to request specific context when needed, making it impossible to measure the efficiency gap between pre-loading and JIT approaches.

Approach

This is a future-focused evaluation initiative designed to inform architectural decisions about context loading strategies in CodeFRAME. The plan follows a phased investigation approach: first audit current behavior to establish baseline metrics, then design and prototype a JIT loading mechanism with tool-based context requests, and finally A/B test both strategies to determine which is optimal. The focus is on measuring context utilization rate (loaded tokens vs. referenced tokens) and time-to-first-token latency. If JIT proves superior, it will be prioritized for implementation; if pre-loading is optimal, the rationale will be documented for future reference. This aligns with Philipp Schmid's context engineering principle of loading context "at the moment" it's needed rather than based on predicted relevance.

Implementation Plan

This evaluation is scoped as a future initiative and should be scheduled after current sprint commitments are complete. The phases below represent the investigation roadmap; implementation of findings will be prioritized based on results.

Phase 1: Audit Current Loading Behavior

1.1 Add Context Utilization Tracking

Objective: Measure how much loaded context is actually used in LLM responses

Changes to file:codeframe/lib/context_manager.py:

• Add new method track_context_utilization(agent_id, loaded_items, llm_response) that:
  • Extracts item IDs/content snippets referenced in LLM response
  • Calculates utilization rate: referenced_tokens / loaded_tokens
  • Stores metrics in new context_utilization_metrics table
  • Returns utilization statistics

Database Schema Addition:

CREATE TABLE context_utilization_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL,
    task_id INTEGER,
    loaded_hot_tokens INTEGER,
    loaded_warm_tokens INTEGER,
    referenced_hot_tokens INTEGER,
    referenced_warm_tokens INTEGER,
    utilization_rate REAL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

Changes to file:codeframe/agents/hybrid_worker.py:

• In execute_task() method after line 196, add tracking call:
  • Pass hot_context, warm_context, and response to utilization tracker
  • Log utilization metrics for analysis

1.2 Trace Tier Promotion Triggers

Objective: Determine if loading is triggered by tier rules or explicit LLM requests

Changes to file:codeframe/lib/context_manager.py:

• Add logging to update_tiers_for_agent() method (line 92) to track:
  • When tier updates occur (periodic vs. event-driven)
  • What triggers tier promotions (age decay, access count, manual)
  • Frequency of COLD → WARM → HOT transitions

Add new method get_tier_transition_history(agent_id, days=7):

• Query context_items table for tier changes over time
• Return transition patterns and frequencies
• Identify if transitions correlate with actual usage

1.3 Measure Baseline Metrics

Objective: Establish current performance baseline

Create new utility file:codeframe/lib/context_metrics.py:

• Method calculate_context_stats(agent_id, project_id) that computes:
  • Average context utilization rate across tasks
  • Token waste per task (loaded but unused)
  • Time-to-first-token for current loading strategy
  • Percentage of WARM items actually referenced in prompts

Add API endpoint in file:codeframe/ui/server.py:

• GET /api/agents/{agent_id}/context/metrics returning:
  • Utilization rate statistics
  • Token efficiency metrics
  • Tier transition patterns

Phase 2: Design JIT Loading Mechanism

2.1 Define Context Request Tool

Objective: Enable agents to request specific context categories on-demand

Create new file file:codeframe/lib/context_tools.py:

• Define request_context_tool function that agents can call:
  • Parameters: category (task_history, test_results, error_logs, code_snippets), filters (time_range, file_path, keywords)
  • Returns: Relevant context items from WARM/COLD tiers
  • Updates access tracking for loaded items

Tool Definition:

{
    "name": "request_context",
    "description": "Load specific context from memory when needed for current task",
    "parameters": {
        "category": ["task_history", "test_results", "error_logs", "code_snippets"],
        "filters": {
            "time_range": "optional ISO date range",
            "file_path": "optional file path pattern",
            "keywords": "optional search terms"
        }
    }
}

Integration with file:codeframe/providers/sdk_client.py:

• Add request_context to available tools list
• Implement tool handler that calls context manager
• Return formatted context to LLM

2.2 Implement Lazy Loading from COLD Storage

Objective: Enable retrieval of archived context when explicitly requested

Changes to file:codeframe/lib/context_manager.py:

• Add method load_cold_context(agent_id, filters) that:
  • Queries archived COLD items matching filters
  • Temporarily promotes to WARM tier for current task
  • Returns items without permanently changing tier
  • Tracks "temporary load" events for analysis

Add checkpoint restoration support:

• Method restore_context_from_checkpoint(checkpoint_id, item_ids) that:
  • Loads specific items from checkpoint JSON
  • Returns items without restoring entire checkpoint
  • Enables selective retrieval of archived context

2.3 Implement Prefetch Hints

Objective: Optimize JIT loading with intelligent prefetching based on task type

Create new file file:codeframe/lib/prefetch_strategy.py:

• Define prefetch rules based on task type:
  • Test tasks → prefetch recent test results and error logs
  • Code tasks → prefetch related file history and dependencies
  • Review tasks → prefetch code changes and quality metrics
• Method get_prefetch_hints(task) returns suggested context categories
• Prefetch happens asynchronously while LLM processes initial prompt

Changes to file:codeframe/agents/hybrid_worker.py:

• In execute_task() method, add prefetch logic after line 162:
  • Get prefetch hints based on task type
  • Asynchronously load suggested WARM items in background
  • Make available if/when agent requests via tool

Phase 3: Prototype JIT Implementation

3.1 Create JIT Worker Agent Variant

Objective: Implement JIT loading strategy for comparison testing

Create new file file:codeframe/agents/jit_worker_agent.py:

• Extend HybridWorkerAgent class
• Override execute_task() method to:
  • Load ONLY HOT tier initially (line 153)
  • Remove pre-loading of WARM tier (delete line 154)
  • Provide request_context tool to SDK client
  • Track when agent uses tool vs. works with HOT only

Modified execution flow:

sequenceDiagram
    participant Agent as JIT Worker Agent
    participant LLM as Claude SDK
    participant CM as Context Manager
    participant DB as Database

    Agent->>CM: load_context(tier=HOT)
    CM->>DB: Query HOT items
    DB-->>CM: Return HOT items
    CM-->>Agent: HOT context only
    
    Agent->>LLM: Execute task with HOT context + request_context tool
    
    alt Agent needs more context
        LLM->>Agent: Call request_context(category="test_results")
        Agent->>CM: load_context_by_category("test_results")
        CM->>DB: Query WARM/COLD items
        DB-->>CM: Return matching items
        CM-->>Agent: Requested context
        Agent->>LLM: Continue with additional context
    end
    
    LLM-->>Agent: Task result
    Agent->>CM: track_utilization(loaded, referenced)

Loading

3.2 Implement A/B Testing Framework

Objective: Compare pre-load vs. JIT strategies with controlled experiments

Create new file file:codeframe/lib/context_experiment.py:

• Class ContextLoadingExperiment that:
  • Randomly assigns tasks to pre-load or JIT strategy
  • Tracks metrics for both approaches
  • Ensures fair comparison (same tasks, same context state)
  • Stores results in context_experiments table

Database Schema:

CREATE TABLE context_experiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    task_id INTEGER NOT NULL,
    strategy TEXT CHECK(strategy IN ('preload', 'jit')),
    tokens_loaded INTEGER,
    tokens_referenced INTEGER,
    utilization_rate REAL,
    time_to_first_token_ms INTEGER,
    task_success BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

Changes to file:codeframe/agents/worker_agent.py:

• Add factory method create_with_strategy(strategy='preload') that:
  • Returns HybridWorkerAgent for preload strategy
  • Returns JITWorkerAgent for JIT strategy
  • Enables easy switching for experiments

3.3 Run Comparative Benchmarks

Objective: Measure performance differences between strategies

Create benchmark script file:scripts/benchmark_context_loading.py:

• Select representative task set (20-30 tasks covering different types)
• Run each task with both strategies
• Measure and compare:
  • Context utilization rate (target: >70% for JIT vs. current)
  • Time-to-first-token (acceptable: <500ms difference)
  • Task success rate (must maintain >90%)
  • Total tokens used per task
  • Number of context requests (JIT only)

Metrics Collection:

• Create dashboard visualization in file:web-ui/src/components/context/ContextExperimentDashboard.tsx
• Display side-by-side comparison:
  • Token efficiency chart
  • Utilization rate distribution
  • Latency comparison
  • Success rate by strategy

Phase 4: Analysis and Decision

4.1 Analyze Experiment Results

Objective: Determine which strategy is optimal based on data

Analysis Criteria:

1. Token Efficiency: JIT should show 20-40% reduction in loaded tokens
2. Latency Impact: JIT overhead should be <300ms per context request
3. Task Success: Both strategies must maintain >90% success rate
4. Context Relevance: JIT should show >70% utilization vs. current ~40-50%

Create analysis report file:docs/context-loading-evaluation.md:

• Document baseline metrics from Phase 1
• Present A/B test results from Phase 3
• Include statistical significance tests
• Provide recommendation with supporting data
• Include cost-benefit analysis for future prioritization

4.2 Implementation Roadmap

If JIT Shows Clear Benefit (>30% token reduction, <200ms latency):

• Create implementation ticket for JIT migration
• Prioritize for next available sprint
• Plan phased rollout: JIT worker agent variant → gradual adoption → full migration
• Update file:specs/007-context-management/spec.md with JIT design

If Pre-loading is Optimal:

• Document rationale in file:docs/context-loading-evaluation.md:
  • Why pre-loading performs better (e.g., latency too high, tool usage unreliable)
  • What optimizations were attempted
  • Under what conditions JIT might be reconsidered
• Create optimization ticket for current approach:
  • Reduce WARM items loaded (currently 20, optimize to 10-15)
  • Improve tier assignment thresholds
  • Add smarter filtering of WARM items

If Hybrid Approach is Best:

• Document hybrid strategy in file:docs/context-loading-evaluation.md
• Create implementation ticket with phased approach:
  • Phase 1: Add prefetch hints for task types
  • Phase 2: Implement selective JIT for COLD tier
  • Phase 3: Optimize WARM tier with on-demand loading
• Plan for future sprint scheduling

Phase 5: Documentation and Findings Report

5.1 Create Comprehensive Evaluation Report

Create file:docs/context-loading-evaluation.md:

• Executive summary with key findings and recommendations
• Baseline metrics from Phase 1 (current utilization rates, token waste)
• A/B test results from Phase 3 with statistical analysis
• Performance comparison charts (token efficiency, latency, success rate)
• Cost-benefit analysis for each strategy
• Recommendations for future implementation
• Reference to Philipp Schmid's context engineering principles

Update file:specs/007-context-management/spec.md:

• Add section "Context Loading Strategy Evaluation Results"
• Document findings and chosen direction
• Include performance metrics and trade-offs
• Link to detailed evaluation report

5.2 Create Future Implementation Tickets

Based on evaluation results, create tickets for:

• If JIT recommended: "Implement JIT Context Loading Strategy"
• If pre-loading optimal: "Optimize Pre-loading Context Strategy"
• If hybrid recommended: "Implement Hybrid Context Loading Strategy"

Each ticket should include:

• Specific implementation steps from this evaluation
• Performance targets and success criteria
• Estimated effort and sprint planning
• Dependencies and risks

5.3 Success Criteria Validation

Verify against ticket requirements:

• ✅ Documented understanding of current loading behavior (Phase 1)
• ✅ Metrics showing context utilization rate (Phase 1.1)
• ✅ Evaluation results with clear recommendation (Phase 4-5)
• ✅ Documented rationale for chosen approach (Phase 5.1)

Evaluation Deliverables:

• Context utilization baseline metrics (current ~40-50%)
• A/B test results comparing pre-load vs. JIT
• Performance analysis (token efficiency, latency, success rate)
• Recommendation with supporting data
• Future implementation roadmap

Key Files Reference

• file:codeframe/agents/hybrid_worker.py - Current pre-loading implementation
• file:codeframe/lib/context_manager.py - Context management logic
• file:codeframe/lib/importance_scorer.py - Tier assignment (note JIT comment line 160)
• file:codeframe/agents/worker_agent.py - Base agent with context methods
• file:codeframe/providers/sdk_client.py - SDK integration for tool support
• file:specs/007-context-management/spec.md - Original context management design

Import In IDE

🤖 Prompt for AI Agents

## Observations

The current CodeFRAME implementation uses a **pre-loading strategy** where both HOT and WARM context tiers are loaded before task execution (`file:codeframe/agents/hybrid_worker.py` lines 153-154). However, comments in `file:codeframe/lib/importance_scorer.py` line 160 indicate WARM tier was designed for "on-demand loading." The system loads all WARM items but only uses the first 20 in prompts (line 226), suggesting significant waste. There's no mechanism to track context utilization rates or enable agents to request specific context when needed, making it impossible to measure the efficiency gap between pre-loading and JIT approaches.

## Approach

This is a **future-focused evaluation initiative** designed to inform architectural decisions about context loading strategies in CodeFRAME. The plan follows a phased investigation approach: first audit current behavior to establish baseline metrics, then design and prototype a JIT loading mechanism with tool-based context requests, and finally A/B test both strategies to determine which is optimal. The focus is on measuring context utilization rate (loaded tokens vs. referenced tokens) and time-to-first-token latency. If JIT proves superior, it will be prioritized for implementation; if pre-loading is optimal, the rationale will be documented for future reference. This aligns with Philipp Schmid's context engineering principle of loading context "at the moment" it's needed rather than based on predicted relevance.

## Implementation Plan

This evaluation is scoped as a **future initiative** and should be scheduled after current sprint commitments are complete. The phases below represent the investigation roadmap; implementation of findings will be prioritized based on results.

### Phase 1: Audit Current Loading Behavior

#### 1.1 Add Context Utilization Tracking

**Objective**: Measure how much loaded context is actually used in LLM responses

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add new method `track_context_utilization(agent_id, loaded_items, llm_response)` that:
  - Extracts item IDs/content snippets referenced in LLM response
  - Calculates utilization rate: `referenced_tokens / loaded_tokens`
  - Stores metrics in new `context_utilization_metrics` table
  - Returns utilization statistics

**Database Schema Addition:**
```sql
CREATE TABLE context_utilization_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL,
    task_id INTEGER,
    loaded_hot_tokens INTEGER,
    loaded_warm_tokens INTEGER,
    referenced_hot_tokens INTEGER,
    referenced_warm_tokens INTEGER,
    utilization_rate REAL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
```

**Changes to `file:codeframe/agents/hybrid_worker.py`:**
- In `execute_task()` method after line 196, add tracking call:
  - Pass `hot_context`, `warm_context`, and `response` to utilization tracker
  - Log utilization metrics for analysis

#### 1.2 Trace Tier Promotion Triggers

**Objective**: Determine if loading is triggered by tier rules or explicit LLM requests

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add logging to `update_tiers_for_agent()` method (line 92) to track:
  - When tier updates occur (periodic vs. event-driven)
  - What triggers tier promotions (age decay, access count, manual)
  - Frequency of COLD → WARM → HOT transitions

**Add new method `get_tier_transition_history(agent_id, days=7)`:**
- Query context_items table for tier changes over time
- Return transition patterns and frequencies
- Identify if transitions correlate with actual usage

#### 1.3 Measure Baseline Metrics

**Objective**: Establish current performance baseline

**Create new utility `file:codeframe/lib/context_metrics.py`:**
- Method `calculate_context_stats(agent_id, project_id)` that computes:
  - Average context utilization rate across tasks
  - Token waste per task (loaded but unused)
  - Time-to-first-token for current loading strategy
  - Percentage of WARM items actually referenced in prompts

**Add API endpoint in `file:codeframe/ui/server.py`:**
- `GET /api/agents/{agent_id}/context/metrics` returning:
  - Utilization rate statistics
  - Token efficiency metrics
  - Tier transition patterns

### Phase 2: Design JIT Loading Mechanism

#### 2.1 Define Context Request Tool

**Objective**: Enable agents to request specific context categories on-demand

**Create new file `file:codeframe/lib/context_tools.py`:**
- Define `request_context_tool` function that agents can call:
  - Parameters: `category` (task_history, test_results, error_logs, code_snippets), `filters` (time_range, file_path, keywords)
  - Returns: Relevant context items from WARM/COLD tiers
  - Updates access tracking for loaded items

**Tool Definition:**
```python
{
    "name": "request_context",
    "description": "Load specific context from memory when needed for current task",
    "parameters": {
        "category": ["task_history", "test_results", "error_logs", "code_snippets"],
        "filters": {
            "time_range": "optional ISO date range",
            "file_path": "optional file path pattern",
            "keywords": "optional search terms"
        }
    }
}
```

**Integration with `file:codeframe/providers/sdk_client.py`:**
- Add `request_context` to available tools list
- Implement tool handler that calls context manager
- Return formatted context to LLM

#### 2.2 Implement Lazy Loading from COLD Storage

**Objective**: Enable retrieval of archived context when explicitly requested

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add method `load_cold_context(agent_id, filters)` that:
  - Queries archived COLD items matching filters
  - Temporarily promotes to WARM tier for current task
  - Returns items without permanently changing tier
  - Tracks "temporary load" events for analysis

**Add checkpoint restoration support:**
- Method `restore_context_from_checkpoint(checkpoint_id, item_ids)` that:
  - Loads specific items from checkpoint JSON
  - Returns items without restoring entire checkpoint
  - Enables selective retrieval of archived context

#### 2.3 Implement Prefetch Hints

**Objective**: Optimize JIT loading with intelligent prefetching based on task type

**Create new file `file:codeframe/lib/prefetch_strategy.py`:**
- Define prefetch rules based on task type:
  - Test tasks → prefetch recent test results and error logs
  - Code tasks → prefetch related file history and dependencies
  - Review tasks → prefetch code changes and quality metrics
- Method `get_prefetch_hints(task)` returns suggested context categories
- Prefetch happens asynchronously while LLM processes initial prompt

**Changes to `file:codeframe/agents/hybrid_worker.py`:**
- In `execute_task()` method, add prefetch logic after line 162:
  - Get prefetch hints based on task type
  - Asynchronously load suggested WARM items in background
  - Make available if/when agent requests via tool

### Phase 3: Prototype JIT Implementation

#### 3.1 Create JIT Worker Agent Variant

**Objective**: Implement JIT loading strategy for comparison testing

**Create new file `file:codeframe/agents/jit_worker_agent.py`:**
- Extend `HybridWorkerAgent` class
- Override `execute_task()` method to:
  - Load ONLY HOT tier initially (line 153)
  - Remove pre-loading of WARM tier (delete line 154)
  - Provide `request_context` tool to SDK client
  - Track when agent uses tool vs. works with HOT only

**Modified execution flow:**
```mermaid
sequenceDiagram
    participant Agent as JIT Worker Agent
    participant LLM as Claude SDK
    participant CM as Context Manager
    participant DB as Database

    Agent->>CM: load_context(tier=HOT)
    CM->>DB: Query HOT items
    DB-->>CM: Return HOT items
    CM-->>Agent: HOT context only
    
    Agent->>LLM: Execute task with HOT context + request_context tool
    
    alt Agent needs more context
        LLM->>Agent: Call request_context(category="test_results")
        Agent->>CM: load_context_by_category("test_results")
        CM->>DB: Query WARM/COLD items
        DB-->>CM: Return matching items
        CM-->>Agent: Requested context
        Agent->>LLM: Continue with additional context
    end
    
    LLM-->>Agent: Task result
    Agent->>CM: track_utilization(loaded, referenced)
```

#### 3.2 Implement A/B Testing Framework

**Objective**: Compare pre-load vs. JIT strategies with controlled experiments

**Create new file `file:codeframe/lib/context_experiment.py`:**
- Class `ContextLoadingExperiment` that:
  - Randomly assigns tasks to pre-load or JIT strategy
  - Tracks metrics for both approaches
  - Ensures fair comparison (same tasks, same context state)
  - Stores results in `context_experiments` table

**Database Schema:**
```sql
CREATE TABLE context_experiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    task_id INTEGER NOT NULL,
    strategy TEXT CHECK(strategy IN ('preload', 'jit')),
    tokens_loaded INTEGER,
    tokens_referenced INTEGER,
    utilization_rate REAL,
    time_to_first_token_ms INTEGER,
    task_success BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
```

**Changes to `file:codeframe/agents/worker_agent.py`:**
- Add factory method `create_with_strategy(strategy='preload')` that:
  - Returns `HybridWorkerAgent` for preload strategy
  - Returns `JITWorkerAgent` for JIT strategy
  - Enables easy switching for experiments

#### 3.3 Run Comparative Benchmarks

**Objective**: Measure performance differences between strategies

**Create benchmark script `file:scripts/benchmark_context_loading.py`:**
- Select representative task set (20-30 tasks covering different types)
- Run each task with both strategies
- Measure and compare:
  - Context utilization rate (target: >70% for JIT vs. current)
  - Time-to-first-token (acceptable: <500ms difference)
  - Task success rate (must maintain >90%)
  - Total tokens used per task
  - Number of context requests (JIT only)

**Metrics Collection:**
- Create dashboard visualization in `file:web-ui/src/components/context/ContextExperimentDashboard.tsx`
- Display side-by-side comparison:
  - Token efficiency chart
  - Utilization rate distribution
  - Latency comparison
  - Success rate by strategy

### Phase 4: Analysis and Decision

#### 4.1 Analyze Experiment Results

**Objective**: Determine which strategy is optimal based on data

**Analysis Criteria:**
1. **Token Efficiency**: JIT should show 20-40% reduction in loaded tokens
2. **Latency Impact**: JIT overhead should be <300ms per context request
3. **Task Success**: Both strategies must maintain >90% success rate
4. **Context Relevance**: JIT should show >70% utilization vs. current ~40-50%

**Create analysis report `file:docs/context-loading-evaluation.md`:**
- Document baseline metrics from Phase 1
- Present A/B test results from Phase 3
- Include statistical significance tests
- Provide recommendation with supporting data
- Include cost-benefit analysis for future prioritization

#### 4.2 Implementation Roadmap

**If JIT Shows Clear Benefit (>30% token reduction, <200ms latency):**
- Create implementation ticket for JIT migration
- Prioritize for next available sprint
- Plan phased rollout: JIT worker agent variant → gradual adoption → full migration
- Update `file:specs/007-context-management/spec.md` with JIT design

**If Pre-loading is Optimal:**
- Document rationale in `file:docs/context-loading-evaluation.md`:
  - Why pre-loading performs better (e.g., latency too high, tool usage unreliable)
  - What optimizations were attempted
  - Under what conditions JIT might be reconsidered
- Create optimization ticket for current approach:
  - Reduce WARM items loaded (currently 20, optimize to 10-15)
  - Improve tier assignment thresholds
  - Add smarter filtering of WARM items

**If Hybrid Approach is Best:**
- Document hybrid strategy in `file:docs/context-loading-evaluation.md`
- Create implementation ticket with phased approach:
  - Phase 1: Add prefetch hints for task types
  - Phase 2: Implement selective JIT for COLD tier
  - Phase 3: Optimize WARM tier with on-demand loading
- Plan for future sprint scheduling

### Phase 5: Documentation and Findings Report

#### 5.1 Create Comprehensive Evaluation Report

**Create `file:docs/context-loading-evaluation.md`:**
- Executive summary with key findings and recommendations
- Baseline metrics from Phase 1 (current utilization rates, token waste)
- A/B test results from Phase 3 with statistical analysis
- Performance comparison charts (token efficiency, latency, success rate)
- Cost-benefit analysis for each strategy
- Recommendations for future implementation
- Reference to Philipp Schmid's context engineering principles

**Update `file:specs/007-context-management/spec.md`:**
- Add section "Context Loading Strategy Evaluation Results"
- Document findings and chosen direction
- Include performance metrics and trade-offs
- Link to detailed evaluation report

#### 5.2 Create Future Implementation Tickets

**Based on evaluation results, create tickets for:**
- If JIT recommended: "Implement JIT Context Loading Strategy"
- If pre-loading optimal: "Optimize Pre-loading Context Strategy"
- If hybrid recommended: "Implement Hybrid Context Loading Strategy"

Each ticket should include:
- Specific implementation steps from this evaluation
- Performance targets and success criteria
- Estimated effort and sprint planning
- Dependencies and risks

#### 5.3 Success Criteria Validation

**Verify against ticket requirements:**
- ✅ Documented understanding of current loading behavior (Phase 1)
- ✅ Metrics showing context utilization rate (Phase 1.1)
- ✅ Evaluation results with clear recommendation (Phase 4-5)
- ✅ Documented rationale for chosen approach (Phase 5.1)

**Evaluation Deliverables:**
- Context utilization baseline metrics (current ~40-50%)
- A/B test results comparing pre-load vs. JIT
- Performance analysis (token efficiency, latency, success rate)
- Recommendation with supporting data
- Future implementation roadmap

## Key Files Reference

- `file:codeframe/agents/hybrid_worker.py` - Current pre-loading implementation
- `file:codeframe/lib/context_manager.py` - Context management logic
- `file:codeframe/lib/importance_scorer.py` - Tier assignment (note JIT comment line 160)
- `file:codeframe/agents/worker_agent.py` - Base agent with context methods
- `file:codeframe/providers/sdk_client.py` - SDK integration for tool support
- `file:specs/007-context-management/spec.md` - Original context management design

---

Execution Information

Branch: main
Commit: 1260477

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[traycerai]

I have updated the Implementation Plan to reflect the recent changes in the ticket.

Previous Plan

Observations

The current CodeFRAME implementation uses a pre-loading strategy where both HOT and WARM context tiers are loaded before task execution (file:codeframe/agents/hybrid_worker.py lines 153-154). However, comments in file:codeframe/lib/importance_scorer.py line 160 indicate WARM tier was designed for "on-demand loading." The system loads all WARM items but only uses the first 20 in prompts (line 226), suggesting significant waste. There's no mechanism to track context utilization rates or enable agents to request specific context when needed, making it impossible to measure the efficiency gap between pre-loading and JIT approaches.

Approach

The plan follows a phased evaluation approach: first audit current behavior to establish baseline metrics, then design and prototype a JIT loading mechanism with tool-based context requests, and finally A/B test both strategies to determine which is optimal. The focus is on measuring context utilization rate (loaded tokens vs. referenced tokens) and time-to-first-token latency. If JIT proves superior, implement it; if pre-loading is optimal, document the rationale. This aligns with Philipp Schmid's context engineering principle of loading context "at the moment" it's needed rather than based on predicted relevance.

Implementation Plan

Phase 1: Audit Current Loading Behavior

1.1 Add Context Utilization Tracking

Objective: Measure how much loaded context is actually used in LLM responses

Changes to file:codeframe/lib/context_manager.py:

• Add new method track_context_utilization(agent_id, loaded_items, llm_response) that:
  • Extracts item IDs/content snippets referenced in LLM response
  • Calculates utilization rate: referenced_tokens / loaded_tokens
  • Stores metrics in new context_utilization_metrics table
  • Returns utilization statistics

Database Schema Addition:

CREATE TABLE context_utilization_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL,
    task_id INTEGER,
    loaded_hot_tokens INTEGER,
    loaded_warm_tokens INTEGER,
    referenced_hot_tokens INTEGER,
    referenced_warm_tokens INTEGER,
    utilization_rate REAL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

Changes to file:codeframe/agents/hybrid_worker.py:

• In execute_task() method after line 196, add tracking call:
  • Pass hot_context, warm_context, and response to utilization tracker
  • Log utilization metrics for analysis

1.2 Trace Tier Promotion Triggers

Objective: Determine if loading is triggered by tier rules or explicit LLM requests

Changes to file:codeframe/lib/context_manager.py:

• Add logging to update_tiers_for_agent() method (line 92) to track:
  • When tier updates occur (periodic vs. event-driven)
  • What triggers tier promotions (age decay, access count, manual)
  • Frequency of COLD → WARM → HOT transitions

Add new method get_tier_transition_history(agent_id, days=7):

• Query context_items table for tier changes over time
• Return transition patterns and frequencies
• Identify if transitions correlate with actual usage

1.3 Measure Baseline Metrics

Objective: Establish current performance baseline

Create new utility file:codeframe/lib/context_metrics.py:

• Method calculate_context_stats(agent_id, project_id) that computes:
  • Average context utilization rate across tasks
  • Token waste per task (loaded but unused)
  • Time-to-first-token for current loading strategy
  • Percentage of WARM items actually referenced in prompts

Add API endpoint in file:codeframe/ui/server.py:

• GET /api/agents/{agent_id}/context/metrics returning:
  • Utilization rate statistics
  • Token efficiency metrics
  • Tier transition patterns

Phase 2: Design JIT Loading Mechanism

2.1 Define Context Request Tool

Objective: Enable agents to request specific context categories on-demand

Create new file file:codeframe/lib/context_tools.py:

• Define request_context_tool function that agents can call:
  • Parameters: category (task_history, test_results, error_logs, code_snippets), filters (time_range, file_path, keywords)
  • Returns: Relevant context items from WARM/COLD tiers
  • Updates access tracking for loaded items

Tool Definition:

{
    "name": "request_context",
    "description": "Load specific context from memory when needed for current task",
    "parameters": {
        "category": ["task_history", "test_results", "error_logs", "code_snippets"],
        "filters": {
            "time_range": "optional ISO date range",
            "file_path": "optional file path pattern",
            "keywords": "optional search terms"
        }
    }
}

Integration with file:codeframe/providers/sdk_client.py:

• Add request_context to available tools list
• Implement tool handler that calls context manager
• Return formatted context to LLM

2.2 Implement Lazy Loading from COLD Storage

Objective: Enable retrieval of archived context when explicitly requested

Changes to file:codeframe/lib/context_manager.py:

• Add method load_cold_context(agent_id, filters) that:
  • Queries archived COLD items matching filters
  • Temporarily promotes to WARM tier for current task
  • Returns items without permanently changing tier
  • Tracks "temporary load" events for analysis

Add checkpoint restoration support:

• Method restore_context_from_checkpoint(checkpoint_id, item_ids) that:
  • Loads specific items from checkpoint JSON
  • Returns items without restoring entire checkpoint
  • Enables selective retrieval of archived context

2.3 Implement Prefetch Hints

Objective: Optimize JIT loading with intelligent prefetching based on task type

Create new file file:codeframe/lib/prefetch_strategy.py:

• Define prefetch rules based on task type:
  • Test tasks → prefetch recent test results and error logs
  • Code tasks → prefetch related file history and dependencies
  • Review tasks → prefetch code changes and quality metrics
• Method get_prefetch_hints(task) returns suggested context categories
• Prefetch happens asynchronously while LLM processes initial prompt

Changes to file:codeframe/agents/hybrid_worker.py:

• In execute_task() method, add prefetch logic after line 162:
  • Get prefetch hints based on task type
  • Asynchronously load suggested WARM items in background
  • Make available if/when agent requests via tool

Phase 3: Prototype JIT Implementation

3.1 Create JIT Worker Agent Variant

Objective: Implement JIT loading strategy for comparison testing

Create new file file:codeframe/agents/jit_worker_agent.py:

• Extend HybridWorkerAgent class
• Override execute_task() method to:
  • Load ONLY HOT tier initially (line 153)
  • Remove pre-loading of WARM tier (delete line 154)
  • Provide request_context tool to SDK client
  • Track when agent uses tool vs. works with HOT only

Modified execution flow:

sequenceDiagram
    participant Agent as JIT Worker Agent
    participant LLM as Claude SDK
    participant CM as Context Manager
    participant DB as Database

    Agent->>CM: load_context(tier=HOT)
    CM->>DB: Query HOT items
    DB-->>CM: Return HOT items
    CM-->>Agent: HOT context only
    
    Agent->>LLM: Execute task with HOT context + request_context tool
    
    alt Agent needs more context
        LLM->>Agent: Call request_context(category="test_results")
        Agent->>CM: load_context_by_category("test_results")
        CM->>DB: Query WARM/COLD items
        DB-->>CM: Return matching items
        CM-->>Agent: Requested context
        Agent->>LLM: Continue with additional context
    end
    
    LLM-->>Agent: Task result
    Agent->>CM: track_utilization(loaded, referenced)

Loading

3.2 Implement A/B Testing Framework

Objective: Compare pre-load vs. JIT strategies with controlled experiments

Create new file file:codeframe/lib/context_experiment.py:

• Class ContextLoadingExperiment that:
  • Randomly assigns tasks to pre-load or JIT strategy
  • Tracks metrics for both approaches
  • Ensures fair comparison (same tasks, same context state)
  • Stores results in context_experiments table

Database Schema:

CREATE TABLE context_experiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    task_id INTEGER NOT NULL,
    strategy TEXT CHECK(strategy IN ('preload', 'jit')),
    tokens_loaded INTEGER,
    tokens_referenced INTEGER,
    utilization_rate REAL,
    time_to_first_token_ms INTEGER,
    task_success BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

Changes to file:codeframe/agents/worker_agent.py:

• Add factory method create_with_strategy(strategy='preload') that:
  • Returns HybridWorkerAgent for preload strategy
  • Returns JITWorkerAgent for JIT strategy
  • Enables easy switching for experiments

3.3 Run Comparative Benchmarks

Objective: Measure performance differences between strategies

Create benchmark script file:scripts/benchmark_context_loading.py:

• Select representative task set (20-30 tasks covering different types)
• Run each task with both strategies
• Measure and compare:
  • Context utilization rate (target: >70% for JIT vs. current)
  • Time-to-first-token (acceptable: <500ms difference)
  • Task success rate (must maintain >90%)
  • Total tokens used per task
  • Number of context requests (JIT only)

Metrics Collection:

• Create dashboard visualization in file:web-ui/src/components/context/ContextExperimentDashboard.tsx
• Display side-by-side comparison:
  • Token efficiency chart
  • Utilization rate distribution
  • Latency comparison
  • Success rate by strategy

Phase 4: Analysis and Decision

4.1 Analyze Experiment Results

Objective: Determine which strategy is optimal based on data

Analysis Criteria:

1. Token Efficiency: JIT should show 20-40% reduction in loaded tokens
2. Latency Impact: JIT overhead should be <300ms per context request
3. Task Success: Both strategies must maintain >90% success rate
4. Context Relevance: JIT should show >70% utilization vs. current ~40-50%

Create analysis report file:docs/context-loading-evaluation.md:

• Document baseline metrics from Phase 1
• Present A/B test results from Phase 3
• Include statistical significance tests
• Provide recommendation with supporting data

4.2 Implementation Decision

If JIT Shows Clear Benefit (>30% token reduction, <200ms latency):

• Replace HybridWorkerAgent implementation with JIT approach
• Update file:codeframe/agents/hybrid_worker.py to use JIT loading
• Add request_context tool to all agent definitions
• Update documentation in file:specs/007-context-management/spec.md

If Pre-loading is Optimal:

• Document rationale in file:docs/context-loading-evaluation.md:
  • Why pre-loading performs better (e.g., latency too high, tool usage unreliable)
  • What optimizations were attempted
  • Under what conditions JIT might be reconsidered
• Optimize current pre-loading approach:
  • Reduce WARM items loaded (currently 20, optimize to 10-15)
  • Improve tier assignment thresholds
  • Add smarter filtering of WARM items

If Hybrid Approach is Best:

• Implement tiered JIT strategy:
  • HOT: Always pre-loaded (critical recent context)
  • WARM: JIT with prefetch hints (on-demand with prediction)
  • COLD: Pure JIT (only when explicitly requested)
• Update file:codeframe/agents/hybrid_worker.py to support hybrid mode
• Add configuration option for loading strategy per agent type

Phase 5: Documentation and Monitoring

5.1 Document Findings

Update file:specs/007-context-management/spec.md:

• Add section "Context Loading Strategy Evaluation"
• Document chosen approach and rationale
• Include performance metrics and trade-offs
• Reference Philipp Schmid's context engineering principles

Create implementation guide file:docs/jit-context-loading.md (if JIT chosen):

• How to use request_context tool in agent prompts
• Best practices for context requests
• Prefetch hint configuration
• Troubleshooting guide

5.2 Add Production Monitoring

Changes to file:codeframe/lib/metrics_tracker.py:

• Add context loading metrics to existing token tracking:
  • Context load time per tier
  • Utilization rate per task
  • Tool usage frequency (if JIT)
  • Cache hit rates (if prefetch enabled)

Dashboard Updates in file:web-ui/src/components/context/ContextPanel.tsx:

• Add real-time utilization rate display
• Show loading strategy indicator (preload/JIT/hybrid)
• Display context request history (if JIT)
• Alert on low utilization rates (<50%)

5.3 Success Criteria Validation

Verify against ticket requirements:

• ✅ Documented understanding of current loading behavior (Phase 1)
• ✅ Metrics showing context utilization rate (Phase 1.1)
• ✅ If JIT beneficial: Implementation with measurable improvement (Phase 3-4)
• ✅ If pre-loading optimal: Documented rationale (Phase 4.2)

Performance Targets:

• Context utilization rate: >70% (vs. current ~40-50%)
• Token efficiency improvement: 20-40% reduction in loaded tokens
• Latency overhead: <300ms for JIT context requests
• Task success rate: Maintain >90%

Key Files Reference

• file:codeframe/agents/hybrid_worker.py - Current pre-loading implementation
• file:codeframe/lib/context_manager.py - Context management logic
• file:codeframe/lib/importance_scorer.py - Tier assignment (note JIT comment line 160)
• file:codeframe/agents/worker_agent.py - Base agent with context methods
• file:codeframe/providers/sdk_client.py - SDK integration for tool support
• file:specs/007-context-management/spec.md - Original context management design

Import In IDE

🤖 Prompt for AI Agents

## Observations

The current CodeFRAME implementation uses a **pre-loading strategy** where both HOT and WARM context tiers are loaded before task execution (`file:codeframe/agents/hybrid_worker.py` lines 153-154). However, comments in `file:codeframe/lib/importance_scorer.py` line 160 indicate WARM tier was designed for "on-demand loading." The system loads all WARM items but only uses the first 20 in prompts (line 226), suggesting significant waste. There's no mechanism to track context utilization rates or enable agents to request specific context when needed, making it impossible to measure the efficiency gap between pre-loading and JIT approaches.

## Approach

The plan follows a phased evaluation approach: first audit current behavior to establish baseline metrics, then design and prototype a JIT loading mechanism with tool-based context requests, and finally A/B test both strategies to determine which is optimal. The focus is on measuring context utilization rate (loaded tokens vs. referenced tokens) and time-to-first-token latency. If JIT proves superior, implement it; if pre-loading is optimal, document the rationale. This aligns with Philipp Schmid's context engineering principle of loading context "at the moment" it's needed rather than based on predicted relevance.

## Implementation Plan

### Phase 1: Audit Current Loading Behavior

#### 1.1 Add Context Utilization Tracking

**Objective**: Measure how much loaded context is actually used in LLM responses

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add new method `track_context_utilization(agent_id, loaded_items, llm_response)` that:
  - Extracts item IDs/content snippets referenced in LLM response
  - Calculates utilization rate: `referenced_tokens / loaded_tokens`
  - Stores metrics in new `context_utilization_metrics` table
  - Returns utilization statistics

**Database Schema Addition:**
```sql
CREATE TABLE context_utilization_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL,
    task_id INTEGER,
    loaded_hot_tokens INTEGER,
    loaded_warm_tokens INTEGER,
    referenced_hot_tokens INTEGER,
    referenced_warm_tokens INTEGER,
    utilization_rate REAL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
```

**Changes to `file:codeframe/agents/hybrid_worker.py`:**
- In `execute_task()` method after line 196, add tracking call:
  - Pass `hot_context`, `warm_context`, and `response` to utilization tracker
  - Log utilization metrics for analysis

#### 1.2 Trace Tier Promotion Triggers

**Objective**: Determine if loading is triggered by tier rules or explicit LLM requests

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add logging to `update_tiers_for_agent()` method (line 92) to track:
  - When tier updates occur (periodic vs. event-driven)
  - What triggers tier promotions (age decay, access count, manual)
  - Frequency of COLD → WARM → HOT transitions

**Add new method `get_tier_transition_history(agent_id, days=7)`:**
- Query context_items table for tier changes over time
- Return transition patterns and frequencies
- Identify if transitions correlate with actual usage

#### 1.3 Measure Baseline Metrics

**Objective**: Establish current performance baseline

**Create new utility `file:codeframe/lib/context_metrics.py`:**
- Method `calculate_context_stats(agent_id, project_id)` that computes:
  - Average context utilization rate across tasks
  - Token waste per task (loaded but unused)
  - Time-to-first-token for current loading strategy
  - Percentage of WARM items actually referenced in prompts

**Add API endpoint in `file:codeframe/ui/server.py`:**
- `GET /api/agents/{agent_id}/context/metrics` returning:
  - Utilization rate statistics
  - Token efficiency metrics
  - Tier transition patterns

### Phase 2: Design JIT Loading Mechanism

#### 2.1 Define Context Request Tool

**Objective**: Enable agents to request specific context categories on-demand

**Create new file `file:codeframe/lib/context_tools.py`:**
- Define `request_context_tool` function that agents can call:
  - Parameters: `category` (task_history, test_results, error_logs, code_snippets), `filters` (time_range, file_path, keywords)
  - Returns: Relevant context items from WARM/COLD tiers
  - Updates access tracking for loaded items

**Tool Definition:**
```python
{
    "name": "request_context",
    "description": "Load specific context from memory when needed for current task",
    "parameters": {
        "category": ["task_history", "test_results", "error_logs", "code_snippets"],
        "filters": {
            "time_range": "optional ISO date range",
            "file_path": "optional file path pattern",
            "keywords": "optional search terms"
        }
    }
}
```

**Integration with `file:codeframe/providers/sdk_client.py`:**
- Add `request_context` to available tools list
- Implement tool handler that calls context manager
- Return formatted context to LLM

#### 2.2 Implement Lazy Loading from COLD Storage

**Objective**: Enable retrieval of archived context when explicitly requested

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add method `load_cold_context(agent_id, filters)` that:
  - Queries archived COLD items matching filters
  - Temporarily promotes to WARM tier for current task
  - Returns items without permanently changing tier
  - Tracks "temporary load" events for analysis

**Add checkpoint restoration support:**
- Method `restore_context_from_checkpoint(checkpoint_id, item_ids)` that:
  - Loads specific items from checkpoint JSON
  - Returns items without restoring entire checkpoint
  - Enables selective retrieval of archived context

#### 2.3 Implement Prefetch Hints

**Objective**: Optimize JIT loading with intelligent prefetching based on task type

**Create new file `file:codeframe/lib/prefetch_strategy.py`:**
- Define prefetch rules based on task type:
  - Test tasks → prefetch recent test results and error logs
  - Code tasks → prefetch related file history and dependencies
  - Review tasks → prefetch code changes and quality metrics
- Method `get_prefetch_hints(task)` returns suggested context categories
- Prefetch happens asynchronously while LLM processes initial prompt

**Changes to `file:codeframe/agents/hybrid_worker.py`:**
- In `execute_task()` method, add prefetch logic after line 162:
  - Get prefetch hints based on task type
  - Asynchronously load suggested WARM items in background
  - Make available if/when agent requests via tool

### Phase 3: Prototype JIT Implementation

#### 3.1 Create JIT Worker Agent Variant

**Objective**: Implement JIT loading strategy for comparison testing

**Create new file `file:codeframe/agents/jit_worker_agent.py`:**
- Extend `HybridWorkerAgent` class
- Override `execute_task()` method to:
  - Load ONLY HOT tier initially (line 153)
  - Remove pre-loading of WARM tier (delete line 154)
  - Provide `request_context` tool to SDK client
  - Track when agent uses tool vs. works with HOT only

**Modified execution flow:**
```mermaid
sequenceDiagram
    participant Agent as JIT Worker Agent
    participant LLM as Claude SDK
    participant CM as Context Manager
    participant DB as Database

    Agent->>CM: load_context(tier=HOT)
    CM->>DB: Query HOT items
    DB-->>CM: Return HOT items
    CM-->>Agent: HOT context only
    
    Agent->>LLM: Execute task with HOT context + request_context tool
    
    alt Agent needs more context
        LLM->>Agent: Call request_context(category="test_results")
        Agent->>CM: load_context_by_category("test_results")
        CM->>DB: Query WARM/COLD items
        DB-->>CM: Return matching items
        CM-->>Agent: Requested context
        Agent->>LLM: Continue with additional context
    end
    
    LLM-->>Agent: Task result
    Agent->>CM: track_utilization(loaded, referenced)
```

#### 3.2 Implement A/B Testing Framework

**Objective**: Compare pre-load vs. JIT strategies with controlled experiments

**Create new file `file:codeframe/lib/context_experiment.py`:**
- Class `ContextLoadingExperiment` that:
  - Randomly assigns tasks to pre-load or JIT strategy
  - Tracks metrics for both approaches
  - Ensures fair comparison (same tasks, same context state)
  - Stores results in `context_experiments` table

**Database Schema:**
```sql
CREATE TABLE context_experiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    task_id INTEGER NOT NULL,
    strategy TEXT CHECK(strategy IN ('preload', 'jit')),
    tokens_loaded INTEGER,
    tokens_referenced INTEGER,
    utilization_rate REAL,
    time_to_first_token_ms INTEGER,
    task_success BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
```

**Changes to `file:codeframe/agents/worker_agent.py`:**
- Add factory method `create_with_strategy(strategy='preload')` that:
  - Returns `HybridWorkerAgent` for preload strategy
  - Returns `JITWorkerAgent` for JIT strategy
  - Enables easy switching for experiments

#### 3.3 Run Comparative Benchmarks

**Objective**: Measure performance differences between strategies

**Create benchmark script `file:scripts/benchmark_context_loading.py`:**
- Select representative task set (20-30 tasks covering different types)
- Run each task with both strategies
- Measure and compare:
  - Context utilization rate (target: >70% for JIT vs. current)
  - Time-to-first-token (acceptable: <500ms difference)
  - Task success rate (must maintain >90%)
  - Total tokens used per task
  - Number of context requests (JIT only)

**Metrics Collection:**
- Create dashboard visualization in `file:web-ui/src/components/context/ContextExperimentDashboard.tsx`
- Display side-by-side comparison:
  - Token efficiency chart
  - Utilization rate distribution
  - Latency comparison
  - Success rate by strategy

### Phase 4: Analysis and Decision

#### 4.1 Analyze Experiment Results

**Objective**: Determine which strategy is optimal based on data

**Analysis Criteria:**
1. **Token Efficiency**: JIT should show 20-40% reduction in loaded tokens
2. **Latency Impact**: JIT overhead should be <300ms per context request
3. **Task Success**: Both strategies must maintain >90% success rate
4. **Context Relevance**: JIT should show >70% utilization vs. current ~40-50%

**Create analysis report `file:docs/context-loading-evaluation.md`:**
- Document baseline metrics from Phase 1
- Present A/B test results from Phase 3
- Include statistical significance tests
- Provide recommendation with supporting data

#### 4.2 Implementation Decision

**If JIT Shows Clear Benefit (>30% token reduction, <200ms latency):**
- Replace `HybridWorkerAgent` implementation with JIT approach
- Update `file:codeframe/agents/hybrid_worker.py` to use JIT loading
- Add `request_context` tool to all agent definitions
- Update documentation in `file:specs/007-context-management/spec.md`

**If Pre-loading is Optimal:**
- Document rationale in `file:docs/context-loading-evaluation.md`:
  - Why pre-loading performs better (e.g., latency too high, tool usage unreliable)
  - What optimizations were attempted
  - Under what conditions JIT might be reconsidered
- Optimize current pre-loading approach:
  - Reduce WARM items loaded (currently 20, optimize to 10-15)
  - Improve tier assignment thresholds
  - Add smarter filtering of WARM items

**If Hybrid Approach is Best:**
- Implement tiered JIT strategy:
  - HOT: Always pre-loaded (critical recent context)
  - WARM: JIT with prefetch hints (on-demand with prediction)
  - COLD: Pure JIT (only when explicitly requested)
- Update `file:codeframe/agents/hybrid_worker.py` to support hybrid mode
- Add configuration option for loading strategy per agent type

### Phase 5: Documentation and Monitoring

#### 5.1 Document Findings

**Update `file:specs/007-context-management/spec.md`:**
- Add section "Context Loading Strategy Evaluation"
- Document chosen approach and rationale
- Include performance metrics and trade-offs
- Reference Philipp Schmid's context engineering principles

**Create implementation guide `file:docs/jit-context-loading.md`** (if JIT chosen):
- How to use `request_context` tool in agent prompts
- Best practices for context requests
- Prefetch hint configuration
- Troubleshooting guide

#### 5.2 Add Production Monitoring

**Changes to `file:codeframe/lib/metrics_tracker.py`:**
- Add context loading metrics to existing token tracking:
  - Context load time per tier
  - Utilization rate per task
  - Tool usage frequency (if JIT)
  - Cache hit rates (if prefetch enabled)

**Dashboard Updates in `file:web-ui/src/components/context/ContextPanel.tsx`:**
- Add real-time utilization rate display
- Show loading strategy indicator (preload/JIT/hybrid)
- Display context request history (if JIT)
- Alert on low utilization rates (<50%)

#### 5.3 Success Criteria Validation

**Verify against ticket requirements:**
- ✅ Documented understanding of current loading behavior (Phase 1)
- ✅ Metrics showing context utilization rate (Phase 1.1)
- ✅ If JIT beneficial: Implementation with measurable improvement (Phase 3-4)
- ✅ If pre-loading optimal: Documented rationale (Phase 4.2)

**Performance Targets:**
- Context utilization rate: >70% (vs. current ~40-50%)
- Token efficiency improvement: 20-40% reduction in loaded tokens
- Latency overhead: <300ms for JIT context requests
- Task success rate: Maintain >90%

## Key Files Reference

- `file:codeframe/agents/hybrid_worker.py` - Current pre-loading implementation
- `file:codeframe/lib/context_manager.py` - Context management logic
- `file:codeframe/lib/importance_scorer.py` - Tier assignment (note JIT comment line 160)
- `file:codeframe/agents/worker_agent.py` - Base agent with context methods
- `file:codeframe/providers/sdk_client.py` - SDK integration for tool support
- `file:specs/007-context-management/spec.md` - Original context management design

---

Execution Information

Branch: main
Commit: 08bbb8e

New Plan

Observations

The current CodeFRAME implementation uses a pre-loading strategy where both HOT and WARM context tiers are loaded before task execution (file:codeframe/agents/hybrid_worker.py lines 153-154). However, comments in file:codeframe/lib/importance_scorer.py line 160 indicate WARM tier was designed for "on-demand loading." The system loads all WARM items but only uses the first 20 in prompts (line 226), suggesting significant waste. There's no mechanism to track context utilization rates or enable agents to request specific context when needed, making it impossible to measure the efficiency gap between pre-loading and JIT approaches.

Approach

This is a future-focused evaluation initiative designed to inform architectural decisions about context loading strategies in CodeFRAME. The plan follows a phased investigation approach: first audit current behavior to establish baseline metrics, then design and prototype a JIT loading mechanism with tool-based context requests, and finally A/B test both strategies to determine which is optimal. The focus is on measuring context utilization rate (loaded tokens vs. referenced tokens) and time-to-first-token latency. If JIT proves superior, it will be prioritized for implementation; if pre-loading is optimal, the rationale will be documented for future reference. This aligns with Philipp Schmid's context engineering principle of loading context "at the moment" it's needed rather than based on predicted relevance.

Implementation Plan

This evaluation is scoped as a future initiative and should be scheduled after current sprint commitments are complete. The phases below represent the investigation roadmap; implementation of findings will be prioritized based on results.

Phase 1: Audit Current Loading Behavior

1.1 Add Context Utilization Tracking

Objective: Measure how much loaded context is actually used in LLM responses

Changes to file:codeframe/lib/context_manager.py:

• Add new method track_context_utilization(agent_id, loaded_items, llm_response) that:
  • Extracts item IDs/content snippets referenced in LLM response
  • Calculates utilization rate: referenced_tokens / loaded_tokens
  • Stores metrics in new context_utilization_metrics table
  • Returns utilization statistics

Database Schema Addition:

CREATE TABLE context_utilization_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL,
    task_id INTEGER,
    loaded_hot_tokens INTEGER,
    loaded_warm_tokens INTEGER,
    referenced_hot_tokens INTEGER,
    referenced_warm_tokens INTEGER,
    utilization_rate REAL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

Changes to file:codeframe/agents/hybrid_worker.py:

• In execute_task() method after line 196, add tracking call:
  • Pass hot_context, warm_context, and response to utilization tracker
  • Log utilization metrics for analysis

1.2 Trace Tier Promotion Triggers

Objective: Determine if loading is triggered by tier rules or explicit LLM requests

Changes to file:codeframe/lib/context_manager.py:

• Add logging to update_tiers_for_agent() method (line 92) to track:
  • When tier updates occur (periodic vs. event-driven)
  • What triggers tier promotions (age decay, access count, manual)
  • Frequency of COLD → WARM → HOT transitions

Add new method get_tier_transition_history(agent_id, days=7):

• Query context_items table for tier changes over time
• Return transition patterns and frequencies
• Identify if transitions correlate with actual usage

1.3 Measure Baseline Metrics

Objective: Establish current performance baseline

Create new utility file:codeframe/lib/context_metrics.py:

• Method calculate_context_stats(agent_id, project_id) that computes:
  • Average context utilization rate across tasks
  • Token waste per task (loaded but unused)
  • Time-to-first-token for current loading strategy
  • Percentage of WARM items actually referenced in prompts

Add API endpoint in file:codeframe/ui/server.py:

• GET /api/agents/{agent_id}/context/metrics returning:
  • Utilization rate statistics
  • Token efficiency metrics
  • Tier transition patterns

Phase 2: Design JIT Loading Mechanism

2.1 Define Context Request Tool

Objective: Enable agents to request specific context categories on-demand

Create new file file:codeframe/lib/context_tools.py:

• Define request_context_tool function that agents can call:
  • Parameters: category (task_history, test_results, error_logs, code_snippets), filters (time_range, file_path, keywords)
  • Returns: Relevant context items from WARM/COLD tiers
  • Updates access tracking for loaded items

Tool Definition:

{
    "name": "request_context",
    "description": "Load specific context from memory when needed for current task",
    "parameters": {
        "category": ["task_history", "test_results", "error_logs", "code_snippets"],
        "filters": {
            "time_range": "optional ISO date range",
            "file_path": "optional file path pattern",
            "keywords": "optional search terms"
        }
    }
}

Integration with file:codeframe/providers/sdk_client.py:

• Add request_context to available tools list
• Implement tool handler that calls context manager
• Return formatted context to LLM

2.2 Implement Lazy Loading from COLD Storage

Objective: Enable retrieval of archived context when explicitly requested

Changes to file:codeframe/lib/context_manager.py:

• Add method load_cold_context(agent_id, filters) that:
  • Queries archived COLD items matching filters
  • Temporarily promotes to WARM tier for current task
  • Returns items without permanently changing tier
  • Tracks "temporary load" events for analysis

Add checkpoint restoration support:

• Method restore_context_from_checkpoint(checkpoint_id, item_ids) that:
  • Loads specific items from checkpoint JSON
  • Returns items without restoring entire checkpoint
  • Enables selective retrieval of archived context

2.3 Implement Prefetch Hints

Objective: Optimize JIT loading with intelligent prefetching based on task type

Create new file file:codeframe/lib/prefetch_strategy.py:

• Define prefetch rules based on task type:
  • Test tasks → prefetch recent test results and error logs
  • Code tasks → prefetch related file history and dependencies
  • Review tasks → prefetch code changes and quality metrics
• Method get_prefetch_hints(task) returns suggested context categories
• Prefetch happens asynchronously while LLM processes initial prompt

Changes to file:codeframe/agents/hybrid_worker.py:

• In execute_task() method, add prefetch logic after line 162:
  • Get prefetch hints based on task type
  • Asynchronously load suggested WARM items in background
  • Make available if/when agent requests via tool

Phase 3: Prototype JIT Implementation

3.1 Create JIT Worker Agent Variant

Objective: Implement JIT loading strategy for comparison testing

Create new file file:codeframe/agents/jit_worker_agent.py:

• Extend HybridWorkerAgent class
• Override execute_task() method to:
  • Load ONLY HOT tier initially (line 153)
  • Remove pre-loading of WARM tier (delete line 154)
  • Provide request_context tool to SDK client
  • Track when agent uses tool vs. works with HOT only

Modified execution flow:

sequenceDiagram
    participant Agent as JIT Worker Agent
    participant LLM as Claude SDK
    participant CM as Context Manager
    participant DB as Database

    Agent->>CM: load_context(tier=HOT)
    CM->>DB: Query HOT items
    DB-->>CM: Return HOT items
    CM-->>Agent: HOT context only
    
    Agent->>LLM: Execute task with HOT context + request_context tool
    
    alt Agent needs more context
        LLM->>Agent: Call request_context(category="test_results")
        Agent->>CM: load_context_by_category("test_results")
        CM->>DB: Query WARM/COLD items
        DB-->>CM: Return matching items
        CM-->>Agent: Requested context
        Agent->>LLM: Continue with additional context
    end
    
    LLM-->>Agent: Task result
    Agent->>CM: track_utilization(loaded, referenced)

Loading

3.2 Implement A/B Testing Framework

Objective: Compare pre-load vs. JIT strategies with controlled experiments

Create new file file:codeframe/lib/context_experiment.py:

• Class ContextLoadingExperiment that:
  • Randomly assigns tasks to pre-load or JIT strategy
  • Tracks metrics for both approaches
  • Ensures fair comparison (same tasks, same context state)
  • Stores results in context_experiments table

Database Schema:

CREATE TABLE context_experiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    task_id INTEGER NOT NULL,
    strategy TEXT CHECK(strategy IN ('preload', 'jit')),
    tokens_loaded INTEGER,
    tokens_referenced INTEGER,
    utilization_rate REAL,
    time_to_first_token_ms INTEGER,
    task_success BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

Changes to file:codeframe/agents/worker_agent.py:

• Add factory method create_with_strategy(strategy='preload') that:
  • Returns HybridWorkerAgent for preload strategy
  • Returns JITWorkerAgent for JIT strategy
  • Enables easy switching for experiments

3.3 Run Comparative Benchmarks

Objective: Measure performance differences between strategies

Create benchmark script file:scripts/benchmark_context_loading.py:

• Select representative task set (20-30 tasks covering different types)
• Run each task with both strategies
• Measure and compare:
  • Context utilization rate (target: >70% for JIT vs. current)
  • Time-to-first-token (acceptable: <500ms difference)
  • Task success rate (must maintain >90%)
  • Total tokens used per task
  • Number of context requests (JIT only)

Metrics Collection:

• Create dashboard visualization in file:web-ui/src/components/context/ContextExperimentDashboard.tsx
• Display side-by-side comparison:
  • Token efficiency chart
  • Utilization rate distribution
  • Latency comparison
  • Success rate by strategy

Phase 4: Analysis and Decision

4.1 Analyze Experiment Results

Objective: Determine which strategy is optimal based on data

Analysis Criteria:

1. Token Efficiency: JIT should show 20-40% reduction in loaded tokens
2. Latency Impact: JIT overhead should be <300ms per context request
3. Task Success: Both strategies must maintain >90% success rate
4. Context Relevance: JIT should show >70% utilization vs. current ~40-50%

Create analysis report file:docs/context-loading-evaluation.md:

• Document baseline metrics from Phase 1
• Present A/B test results from Phase 3
• Include statistical significance tests
• Provide recommendation with supporting data
• Include cost-benefit analysis for future prioritization

4.2 Implementation Roadmap

If JIT Shows Clear Benefit (>30% token reduction, <200ms latency):

• Create implementation ticket for JIT migration
• Prioritize for next available sprint
• Plan phased rollout: JIT worker agent variant → gradual adoption → full migration
• Update file:specs/007-context-management/spec.md with JIT design

If Pre-loading is Optimal:

• Document rationale in file:docs/context-loading-evaluation.md:
  • Why pre-loading performs better (e.g., latency too high, tool usage unreliable)
  • What optimizations were attempted
  • Under what conditions JIT might be reconsidered
• Create optimization ticket for current approach:
  • Reduce WARM items loaded (currently 20, optimize to 10-15)
  • Improve tier assignment thresholds
  • Add smarter filtering of WARM items

If Hybrid Approach is Best:

• Document hybrid strategy in file:docs/context-loading-evaluation.md
• Create implementation ticket with phased approach:
  • Phase 1: Add prefetch hints for task types
  • Phase 2: Implement selective JIT for COLD tier
  • Phase 3: Optimize WARM tier with on-demand loading
• Plan for future sprint scheduling

Phase 5: Documentation and Findings Report

5.1 Create Comprehensive Evaluation Report

Create file:docs/context-loading-evaluation.md:

• Executive summary with key findings and recommendations
• Baseline metrics from Phase 1 (current utilization rates, token waste)
• A/B test results from Phase 3 with statistical analysis
• Performance comparison charts (token efficiency, latency, success rate)
• Cost-benefit analysis for each strategy
• Recommendations for future implementation
• Reference to Philipp Schmid's context engineering principles

Update file:specs/007-context-management/spec.md:

• Add section "Context Loading Strategy Evaluation Results"
• Document findings and chosen direction
• Include performance metrics and trade-offs
• Link to detailed evaluation report

5.2 Create Future Implementation Tickets

Based on evaluation results, create tickets for:

• If JIT recommended: "Implement JIT Context Loading Strategy"
• If pre-loading optimal: "Optimize Pre-loading Context Strategy"
• If hybrid recommended: "Implement Hybrid Context Loading Strategy"

Each ticket should include:

• Specific implementation steps from this evaluation
• Performance targets and success criteria
• Estimated effort and sprint planning
• Dependencies and risks

5.3 Success Criteria Validation

Verify against ticket requirements:

• ✅ Documented understanding of current loading behavior (Phase 1)
• ✅ Metrics showing context utilization rate (Phase 1.1)
• ✅ Evaluation results with clear recommendation (Phase 4-5)
• ✅ Documented rationale for chosen approach (Phase 5.1)

Evaluation Deliverables:

• Context utilization baseline metrics (current ~40-50%)
• A/B test results comparing pre-load vs. JIT
• Performance analysis (token efficiency, latency, success rate)
• Recommendation with supporting data
• Future implementation roadmap

Key Files Reference

• file:codeframe/agents/hybrid_worker.py - Current pre-loading implementation
• file:codeframe/lib/context_manager.py - Context management logic
• file:codeframe/lib/importance_scorer.py - Tier assignment (note JIT comment line 160)
• file:codeframe/agents/worker_agent.py - Base agent with context methods
• file:codeframe/providers/sdk_client.py - SDK integration for tool support
• file:specs/007-context-management/spec.md - Original context management design

Import In IDE

🤖 Prompt for AI Agents

## Observations

The current CodeFRAME implementation uses a **pre-loading strategy** where both HOT and WARM context tiers are loaded before task execution (`file:codeframe/agents/hybrid_worker.py` lines 153-154). However, comments in `file:codeframe/lib/importance_scorer.py` line 160 indicate WARM tier was designed for "on-demand loading." The system loads all WARM items but only uses the first 20 in prompts (line 226), suggesting significant waste. There's no mechanism to track context utilization rates or enable agents to request specific context when needed, making it impossible to measure the efficiency gap between pre-loading and JIT approaches.

## Approach

This is a **future-focused evaluation initiative** designed to inform architectural decisions about context loading strategies in CodeFRAME. The plan follows a phased investigation approach: first audit current behavior to establish baseline metrics, then design and prototype a JIT loading mechanism with tool-based context requests, and finally A/B test both strategies to determine which is optimal. The focus is on measuring context utilization rate (loaded tokens vs. referenced tokens) and time-to-first-token latency. If JIT proves superior, it will be prioritized for implementation; if pre-loading is optimal, the rationale will be documented for future reference. This aligns with Philipp Schmid's context engineering principle of loading context "at the moment" it's needed rather than based on predicted relevance.

## Implementation Plan

This evaluation is scoped as a **future initiative** and should be scheduled after current sprint commitments are complete. The phases below represent the investigation roadmap; implementation of findings will be prioritized based on results.

### Phase 1: Audit Current Loading Behavior

#### 1.1 Add Context Utilization Tracking

**Objective**: Measure how much loaded context is actually used in LLM responses

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add new method `track_context_utilization(agent_id, loaded_items, llm_response)` that:
  - Extracts item IDs/content snippets referenced in LLM response
  - Calculates utilization rate: `referenced_tokens / loaded_tokens`
  - Stores metrics in new `context_utilization_metrics` table
  - Returns utilization statistics

**Database Schema Addition:**
```sql
CREATE TABLE context_utilization_metrics (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL,
    task_id INTEGER,
    loaded_hot_tokens INTEGER,
    loaded_warm_tokens INTEGER,
    referenced_hot_tokens INTEGER,
    referenced_warm_tokens INTEGER,
    utilization_rate REAL,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
```

**Changes to `file:codeframe/agents/hybrid_worker.py`:**
- In `execute_task()` method after line 196, add tracking call:
  - Pass `hot_context`, `warm_context`, and `response` to utilization tracker
  - Log utilization metrics for analysis

#### 1.2 Trace Tier Promotion Triggers

**Objective**: Determine if loading is triggered by tier rules or explicit LLM requests

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add logging to `update_tiers_for_agent()` method (line 92) to track:
  - When tier updates occur (periodic vs. event-driven)
  - What triggers tier promotions (age decay, access count, manual)
  - Frequency of COLD → WARM → HOT transitions

**Add new method `get_tier_transition_history(agent_id, days=7)`:**
- Query context_items table for tier changes over time
- Return transition patterns and frequencies
- Identify if transitions correlate with actual usage

#### 1.3 Measure Baseline Metrics

**Objective**: Establish current performance baseline

**Create new utility `file:codeframe/lib/context_metrics.py`:**
- Method `calculate_context_stats(agent_id, project_id)` that computes:
  - Average context utilization rate across tasks
  - Token waste per task (loaded but unused)
  - Time-to-first-token for current loading strategy
  - Percentage of WARM items actually referenced in prompts

**Add API endpoint in `file:codeframe/ui/server.py`:**
- `GET /api/agents/{agent_id}/context/metrics` returning:
  - Utilization rate statistics
  - Token efficiency metrics
  - Tier transition patterns

### Phase 2: Design JIT Loading Mechanism

#### 2.1 Define Context Request Tool

**Objective**: Enable agents to request specific context categories on-demand

**Create new file `file:codeframe/lib/context_tools.py`:**
- Define `request_context_tool` function that agents can call:
  - Parameters: `category` (task_history, test_results, error_logs, code_snippets), `filters` (time_range, file_path, keywords)
  - Returns: Relevant context items from WARM/COLD tiers
  - Updates access tracking for loaded items

**Tool Definition:**
```python
{
    "name": "request_context",
    "description": "Load specific context from memory when needed for current task",
    "parameters": {
        "category": ["task_history", "test_results", "error_logs", "code_snippets"],
        "filters": {
            "time_range": "optional ISO date range",
            "file_path": "optional file path pattern",
            "keywords": "optional search terms"
        }
    }
}
```

**Integration with `file:codeframe/providers/sdk_client.py`:**
- Add `request_context` to available tools list
- Implement tool handler that calls context manager
- Return formatted context to LLM

#### 2.2 Implement Lazy Loading from COLD Storage

**Objective**: Enable retrieval of archived context when explicitly requested

**Changes to `file:codeframe/lib/context_manager.py`:**
- Add method `load_cold_context(agent_id, filters)` that:
  - Queries archived COLD items matching filters
  - Temporarily promotes to WARM tier for current task
  - Returns items without permanently changing tier
  - Tracks "temporary load" events for analysis

**Add checkpoint restoration support:**
- Method `restore_context_from_checkpoint(checkpoint_id, item_ids)` that:
  - Loads specific items from checkpoint JSON
  - Returns items without restoring entire checkpoint
  - Enables selective retrieval of archived context

#### 2.3 Implement Prefetch Hints

**Objective**: Optimize JIT loading with intelligent prefetching based on task type

**Create new file `file:codeframe/lib/prefetch_strategy.py`:**
- Define prefetch rules based on task type:
  - Test tasks → prefetch recent test results and error logs
  - Code tasks → prefetch related file history and dependencies
  - Review tasks → prefetch code changes and quality metrics
- Method `get_prefetch_hints(task)` returns suggested context categories
- Prefetch happens asynchronously while LLM processes initial prompt

**Changes to `file:codeframe/agents/hybrid_worker.py`:**
- In `execute_task()` method, add prefetch logic after line 162:
  - Get prefetch hints based on task type
  - Asynchronously load suggested WARM items in background
  - Make available if/when agent requests via tool

### Phase 3: Prototype JIT Implementation

#### 3.1 Create JIT Worker Agent Variant

**Objective**: Implement JIT loading strategy for comparison testing

**Create new file `file:codeframe/agents/jit_worker_agent.py`:**
- Extend `HybridWorkerAgent` class
- Override `execute_task()` method to:
  - Load ONLY HOT tier initially (line 153)
  - Remove pre-loading of WARM tier (delete line 154)
  - Provide `request_context` tool to SDK client
  - Track when agent uses tool vs. works with HOT only

**Modified execution flow:**
```mermaid
sequenceDiagram
    participant Agent as JIT Worker Agent
    participant LLM as Claude SDK
    participant CM as Context Manager
    participant DB as Database

    Agent->>CM: load_context(tier=HOT)
    CM->>DB: Query HOT items
    DB-->>CM: Return HOT items
    CM-->>Agent: HOT context only
    
    Agent->>LLM: Execute task with HOT context + request_context tool
    
    alt Agent needs more context
        LLM->>Agent: Call request_context(category="test_results")
        Agent->>CM: load_context_by_category("test_results")
        CM->>DB: Query WARM/COLD items
        DB-->>CM: Return matching items
        CM-->>Agent: Requested context
        Agent->>LLM: Continue with additional context
    end
    
    LLM-->>Agent: Task result
    Agent->>CM: track_utilization(loaded, referenced)
```

#### 3.2 Implement A/B Testing Framework

**Objective**: Compare pre-load vs. JIT strategies with controlled experiments

**Create new file `file:codeframe/lib/context_experiment.py`:**
- Class `ContextLoadingExperiment` that:
  - Randomly assigns tasks to pre-load or JIT strategy
  - Tracks metrics for both approaches
  - Ensures fair comparison (same tasks, same context state)
  - Stores results in `context_experiments` table

**Database Schema:**
```sql
CREATE TABLE context_experiments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    task_id INTEGER NOT NULL,
    strategy TEXT CHECK(strategy IN ('preload', 'jit')),
    tokens_loaded INTEGER,
    tokens_referenced INTEGER,
    utilization_rate REAL,
    time_to_first_token_ms INTEGER,
    task_success BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
```

**Changes to `file:codeframe/agents/worker_agent.py`:**
- Add factory method `create_with_strategy(strategy='preload')` that:
  - Returns `HybridWorkerAgent` for preload strategy
  - Returns `JITWorkerAgent` for JIT strategy
  - Enables easy switching for experiments

#### 3.3 Run Comparative Benchmarks

**Objective**: Measure performance differences between strategies

**Create benchmark script `file:scripts/benchmark_context_loading.py`:**
- Select representative task set (20-30 tasks covering different types)
- Run each task with both strategies
- Measure and compare:
  - Context utilization rate (target: >70% for JIT vs. current)
  - Time-to-first-token (acceptable: <500ms difference)
  - Task success rate (must maintain >90%)
  - Total tokens used per task
  - Number of context requests (JIT only)

**Metrics Collection:**
- Create dashboard visualization in `file:web-ui/src/components/context/ContextExperimentDashboard.tsx`
- Display side-by-side comparison:
  - Token efficiency chart
  - Utilization rate distribution
  - Latency comparison
  - Success rate by strategy

### Phase 4: Analysis and Decision

#### 4.1 Analyze Experiment Results

**Objective**: Determine which strategy is optimal based on data

**Analysis Criteria:**
1. **Token Efficiency**: JIT should show 20-40% reduction in loaded tokens
2. **Latency Impact**: JIT overhead should be <300ms per context request
3. **Task Success**: Both strategies must maintain >90% success rate
4. **Context Relevance**: JIT should show >70% utilization vs. current ~40-50%

**Create analysis report `file:docs/context-loading-evaluation.md`:**
- Document baseline metrics from Phase 1
- Present A/B test results from Phase 3
- Include statistical significance tests
- Provide recommendation with supporting data
- Include cost-benefit analysis for future prioritization

#### 4.2 Implementation Roadmap

**If JIT Shows Clear Benefit (>30% token reduction, <200ms latency):**
- Create implementation ticket for JIT migration
- Prioritize for next available sprint
- Plan phased rollout: JIT worker agent variant → gradual adoption → full migration
- Update `file:specs/007-context-management/spec.md` with JIT design

**If Pre-loading is Optimal:**
- Document rationale in `file:docs/context-loading-evaluation.md`:
  - Why pre-loading performs better (e.g., latency too high, tool usage unreliable)
  - What optimizations were attempted
  - Under what conditions JIT might be reconsidered
- Create optimization ticket for current approach:
  - Reduce WARM items loaded (currently 20, optimize to 10-15)
  - Improve tier assignment thresholds
  - Add smarter filtering of WARM items

**If Hybrid Approach is Best:**
- Document hybrid strategy in `file:docs/context-loading-evaluation.md`
- Create implementation ticket with phased approach:
  - Phase 1: Add prefetch hints for task types
  - Phase 2: Implement selective JIT for COLD tier
  - Phase 3: Optimize WARM tier with on-demand loading
- Plan for future sprint scheduling

### Phase 5: Documentation and Findings Report

#### 5.1 Create Comprehensive Evaluation Report

**Create `file:docs/context-loading-evaluation.md`:**
- Executive summary with key findings and recommendations
- Baseline metrics from Phase 1 (current utilization rates, token waste)
- A/B test results from Phase 3 with statistical analysis
- Performance comparison charts (token efficiency, latency, success rate)
- Cost-benefit analysis for each strategy
- Recommendations for future implementation
- Reference to Philipp Schmid's context engineering principles

**Update `file:specs/007-context-management/spec.md`:**
- Add section "Context Loading Strategy Evaluation Results"
- Document findings and chosen direction
- Include performance metrics and trade-offs
- Link to detailed evaluation report

#### 5.2 Create Future Implementation Tickets

**Based on evaluation results, create tickets for:**
- If JIT recommended: "Implement JIT Context Loading Strategy"
- If pre-loading optimal: "Optimize Pre-loading Context Strategy"
- If hybrid recommended: "Implement Hybrid Context Loading Strategy"

Each ticket should include:
- Specific implementation steps from this evaluation
- Performance targets and success criteria
- Estimated effort and sprint planning
- Dependencies and risks

#### 5.3 Success Criteria Validation

**Verify against ticket requirements:**
- ✅ Documented understanding of current loading behavior (Phase 1)
- ✅ Metrics showing context utilization rate (Phase 1.1)
- ✅ Evaluation results with clear recommendation (Phase 4-5)
- ✅ Documented rationale for chosen approach (Phase 5.1)

**Evaluation Deliverables:**
- Context utilization baseline metrics (current ~40-50%)
- A/B test results comparing pre-load vs. JIT
- Performance analysis (token efficiency, latency, success rate)
- Recommendation with supporting data
- Future implementation roadmap

## Key Files Reference

- `file:codeframe/agents/hybrid_worker.py` - Current pre-loading implementation
- `file:codeframe/lib/context_manager.py` - Context management logic
- `file:codeframe/lib/importance_scorer.py` - Tier assignment (note JIT comment line 160)
- `file:codeframe/agents/worker_agent.py` - Base agent with context methods
- `file:codeframe/providers/sdk_client.py` - SDK integration for tool support
- `file:specs/007-context-management/spec.md` - Original context management design

---

Execution Information

Branch: main
Commit: 1260477